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ABSTRACT 

This report focuses on the extent to which 
conclusions about the effectiveness of compensatory education . 
programs are affected by (1) the period* of time on which the 
evaluation is based, and (2y the standard against which the program 
is judged. Previously collected evaluation data frci four 
compensatory education prcarans (grades 1-8) are reanalyzed in order 
to test, the effects of these ccmponents. Program effectiveness is 
Judged according to the standards of both grade-eguivalent and 
percentile point achievement in^^ices, as measured by stemdardi^zed 
tests.. The primary .finding of these analyses is -that cpnclusions 
about program effectiveness are greatly influenced by the the period 
of time over which the evaluation is based. Spedif ically,^ it is shown 
that the inclus5.on of the' summer months in the evaluation can reduce 
estiaStes 6t achievement and reverse positive judgments of program 
effectiveness. Additionally, it is demoTistrated that the 10 
percentile point achievement standard. is more Stringent than . 
grade-eguivalent score sti^.dards, and is thus less likely-^' be met 
in a 12-ionth evaluation. Based on this study's fin'dirgs, it, is 
recommended thfit evaluation of compensatory programs be based On a 
12-month (fall to fall), period and that students npt be: promoted on 
the basis of spring test scores. (Author/GC) . 
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' i INTRODUCTION ' , . 

Evaluations of compensatory education programs have raised more 
questvions than ^they have ahswered* Much of the resulting confusion is' 
inevitalSle in a field that is young and expanding. The confusion is 
'Exacerbated, however, by a lack of adequate attention to how program 
effectiveness is defined, particularly in the context of the philosophy 
of compensatory education. This study Is an attempt to clarify some-of 
the issues involved in defining the effectiveness of compensatory educa- 
tion programs. . , • . 

The work reported here concerns the extent to which conclusions 
about the effectiveness of compensatory ^education programs are affected 
by two major components of an evaluation: the period of time on which 
the evaluation is based and the standard against which the progr^'s 
effectiveness is judged ♦ We argue in particular that the philosophy of 
compensatorjH education suggests that evaluations should measure program 
effectiveness over a period of time longer than the school-year; in other 
words, that evaluations should assess the extent to which effects'* -ere 
sustained. Therefore, we calculate achievement gains for several programs 
based on at least two periods of time: <the traditional fall pretest to 
spring posttest (school-year) evaluat;ion period and a 12-month, f ati- 
to-fall period that includes the summer following the program. 

We then draw conclusions about program effectiveness "based on three 
standards for success and compare the conclusions for the different time 
periods. These standards are derived from/ those previously used in 
evaluations of compensatory education programs and use the norms of 
standardized tests as the frame of reference. Two of the standards are 
expressed in the metric! of grade equivalents: - a rate of gain of one 
grade-equivalent month for each month in the program and an annual rate 
of 8 months* The third standard is a gain of ten percentile points. 
In th6 absence 6^ information on the expected achievement of disadvantaged 

1 , 



students without compensatory education experience, we do nojt select a 



"best" standard, but rather demonstrate the, extent to which conclusions 
abqut effectiveness differ according to tl^e standard and the period of 
time used. 

* ■ 

\ Section II provides an extended discussion* of the rationale for 
measuring sxistained effectiveness and for our choice of standards. In 
Section III, we present a description of the search for appropriate d^ta 
and each data set obtained. Section IV contains the results of th.e 
primary analyses. Section V presents supplemental analyses, and our 
conclusions ' are presented and- discussed in Section VI. 

To simplify the presentation in the text, we have relegated a large 
number of tables artd detail to appendices which are bound, separately. 
This material is referenced throughout the text* 
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i ^ ... 

Research on vcompensatory education programs has failed to product a 

widely^/ accepted definition of program effectiveness. In fact, research 
and evaluation are rarely conducted with a clear definition of "^u^ces^." 
Researchers and practitioners define effectiveness in a number o^^ays/ 
ranging from vague statements such as "better than expected" to more 
sophisticated statements of a required magnitude of change. The purpose 
of this work is not to develop a single definition of effectiveness, but 
to demonstrate how different definitions of effectiveness can lead to 
different conclu3ions about program success or failure. Instead of de-- 
veloping a specific definition with limited application., we specify the 
major ingredients^necessary for a definition. In particular, we concen- 
trate on two major components of a definition of effectiveness: tl^ 
perxjpd of time on. which the evaluation is based and the standard against j 
which the program is judged. 



We begin with the assertion that effectiveness should be defined in 

the context of the goals of cohipensatory education programs, and that 

these goals should determine what should be njeasured and when the measure-^ 

ments should occur. We have chosen to restrict ourselves to one freq4iently 

r 

stated and often measured goal of compensatory education, the improvement ' 
of cognitive skills as measured by standardized achievement tests. Itv 
the remainder of this section, we discuss the period of ^ime on which an 
evaluation is based and the standards that we' employ in judging program 
effectiveness. - 



Period of Time 



A fundamental assumption of compensatory education is that greater 
^achievement can change the ac^emic future of disadvantaged students, 
in turn enhancing their "life chances." Therefore, one of the goals of 
compensatory education is to increase the achievement of disadvantaged 
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■students. In order . to. improve^students * futures this Increase in ^ * 

•achievement sh^tj^Id be evident subsequent to - participation in a campertsa- 

,to1:y- education program. At a minimum, an increase In achievement should 

persi^» ovgr the summer following a school-year program. However," evalua-^ 

tions of compensatory education in general, ajncj of Title I of the' Ele- 

mentary and Secondary Education !\ct (ESEA) in particular, have not - 

included estimates of sustained achievement. Instead, judgments of 
«» • . - ' ^ 

program success have been based on Students* achievement during the* 

school year; that is, on a spring posttest score adjusted in some way 

for the preceding fall pretest score. • " . . y 

^ We hypothesized that evaluations based on measures of sustained • 
achievement would lead to different conclusions . than evaluations' based* 
on school-year (fall-to-spring) achievement. Specif icalTy, we hypothe- 
sized that evaluations based on a fall-to-fall period, by virtue of 
including the summer months-, would result in smaller achievement gains 
than traditional school-year evaluations. We were led to this hypothesis 
in part by studies that compare the achievement rates of disa^dvantaged 
students during the school year and during the summer (Hayes and Grether, 
1969; Heyns, 1976; Murnane, 1975). These studies, while extremely limited 
present some evidence that disadvantaged students achieve at a slower 

s 

rate than expected over the summer. Both conventional wisdom and the 
standardization procedures of achievement tests assume that the rate of 
achievement for all students is slower during the summer than during the 
school year. The grade-equivalent scale defines the rate of achievement 
of the 50th percentile student as 9 months over the 9-month school year 
and 1 month over the 3-month summer. Hence the. summer r^te is assumed- 
to te one- third' the school-year rate. This pattern of Achievemeat is 
^pJ^sumed to be the same for both advantaged and disadvantaged students: 
all students are assumed to gain over the summer but at a slower rate 
than over, the scl^ool year. The studies ciM:ed above suggest that this 
is not the case for disadvantaged students. In fact, disadvantaged 
students may have no gain over the summer or tmy even lose. 

The development of the hypothesis was also influenced by the fact 
that evidence of success of Title I students during the school year was 



not supported by other sources of data. Specifically, State Title I * 
evaluations show that students in Title I t>r(^grains achieve at much highen, 
rates than expected during the school year.* This finding is not 
supported, however, by data from statewide testing programs. Since the , 
advent of .Title I, there are ho detectable Increases iif^ the scores of 
.those most likely to be Title I participants — the low-percentile stud^hts 
(Thomas and Pelavin, 1976). . • • 

Together, these findings suggest that large achievement gains 
produced by compensatory^education programs over the school 'year may be 
followed by corresponding achievement losses over the summer. If such 
summer lofeses occur, whether or not they are proportional to school-year 
gains, evaluations including the summer months will result in smaller 
achievement gains" than evaluations based on the traditional fall-to-spring 
time period. 

Consequently, one major goal of our study was to compare achievement 

gains for several pro'grams based on different periods of time. However,^ 

the period of time used in an evaluation is not the only component that 

determines whether or not a program is effective. There must^also be 

a standard against which achievement gains are judged. Therefore, tJte 

second goal of our^ work was to illustrate the extent to which conclusions 

about program effectiveness are affected by the standard used. The 

stan4^rds that we applied and the rationale for using them are described 

below. . > ' - " 

ft' ^ 

Three Standards for Succe ss 

A major . problem in the evaluation of compensatory education programs 
is the lack of information oYi the expected achievement of disadvantaged 
students not participating in compensatory education. To determine what 
portion of an achievement gain is directly attributable to a compensatory 



*The expectation is based on the unofficial Title I standard for success, 
'■•which is one grade-equivalent month gained for each month in the program/ 



education program, the evaluator must have some notion of what would 
have happened to students^ achievement had they not participated in the 
program. There is not a large body of data on educationally' disadvantaged' 
students who have not_ been in compensatory education programs • And as 
more educationally disadvantage^ students participate in compensatory 
progranjs, such ''baseline" ^ata become more difficult to obtain. In the 
absence of such baseline data, evaluators are faced with A choice among 
seve:ral less than satisfactory alternatives such as using various types 
of "control" groups or using the'norms of standardised tests as the frame 
of reference . ' - . - 

^ In evaluations of Title I programs, the use of standards derived 

..from standardized test n^rms is by far the most common approach. This 
is partly because the standards, which are expressed in grade-equivalents 
or percentiles, can be applied across different tests and thus used in 
aggregating data for national purposes. One such standard that has been^ 
applied in the past by the U.S-. Office of Education (US(^) is an average 
achievement rate of one grad?-equivalent month per month during the 
school year. A second is a variation of tH,e standard suggested in one 
of the recently adopted 'USO^ evaluation models: a percentile increase 
equivalent to one-third of the standard deviation of the norm group. A 
third, in the language of grade-equivalents, is in fact empirically 
based: an achievement gain of 8 grade-equivalent months. The genesis 
and characteristics of each of these standards are discussed below, 

Month-for-Month Sta nd^fd 

Procedures for developing the grade-equivalent scale vary somewhf^t\^ 
from one test publisher to anptheV, but all tests define the achievement 
rate for the average or 50th percentile student to be one grade-equivalent 
month pe^ month during the 9-month school year and one grade-equivalent 
month over the 3-month summfer* The month-f or-moath standard stems from 
this ach:^vement rate, and its application to compensatory education . 
p^rograms rests on the assumption'* that a disadvantaged student achieving 
at the rate of the 50th percentile student is doing better tKan expected. 
To demonstrate that this assumption is ht least open to question, we 
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describe In dverslmpllf led fashion the derivation of the grade-equivalent 
. scale. 

A standardized achievement test Is riot one test but a bdttery con- 
slstlng of several test levels, each spanning one or more grades. The 
)ioYmlng of the b.Attery consists of^ administering adjacent levels of tt{e 
test battery In ea6h grade to a sample considered to be nationally repre-"* 
sentatlye,. From tiles;^^ raw scores, a scale Is developed^ spanning all 

test levels , ^that alla^^s tr*tartslfitlon of eaeh raw score Into a alngle ' - 

; - , ^' ' ' 

metric. The median* scorp at each grade G Is assigned the grade-equivalent 

jcore of G.X where X Is the number of the month of the school year In | 

which the test was standardized. For example. If the test were admlnis- 

tered In October (one month Into the school year) , the median score for 
■ • , * 

third graders would be assigned a grade-equivalent score of 3.1. By 

r- 

assigning 'the Appreciate grade-eqlilvalent score to the median score at " 

<• , » 

each grade, a set of grade-equivalent scores (1.1, 2.1, 3.1, etc.) can 
be plotted against the scale scopes that span all levels of the test. 
In essence, the omitted '.^ade-equlvalents (3.2, 3.3, 3. A, etc.) are 
Interpolated by dividing the distance between consecfutlve median ^scores 
into tenths. ThuB the score that is one-tenth of the distance from- the 
third grade medlan*lB??ore of 3.1 to the fourth grade median of A.l is 
assigned the value of 3.2, and so on. 

Both the development of a scale that spans test levels and the 
Interpolat^jLons between median .scores entail quite complex mathematical 
manipulations from the application of Thursfone scaling techniques to 
the fitting of high-order polynomials. The above description is Intended 
only to prdvide a sketch of the development of the grade-equivalent scaj^e^ 
with the understanding that the actual procedures are quite complicated 
and vary from test to test. * 

The point of ^describing this procedure is to provide At understanding 

I 

of the empirical basis for the month-f or-month standard. In essence, the 
pattern of achievement described by the grade-equivalent , scale is based 



This discussioh describes standardization procedures based on one test 
administration. 



only on median, scoresV a grade-equivalent year apart. All other grade- 
equivaleAtl ""are ^^stimated through Intettpolat ion , It is important to 
Vecognize the features of this process. First, the pattern of growth 
.ascribed to the average student is arbitrarily defined to be 1 month *s / 
growth per month' and is anchored in reality at only^one point — the mdtr^ ' 
of thfe school *yea^ in which the test was standardized • Second , there ; 
is no empirical information on the pattero of achievement for low-achieving 
{of hlgh-aehieving) students. The one empirical point is based, bnly on . 
tKe 50th percentile student, Henpe, the ^ssumpt^on that this overall 



patterh. holds for pppils pther thah t\ie average student has .little 
empirical basis^ 



A small and growing number of tests are Aormed on the basis of two 
test administrations — one in the fall and one^ in the ^prlng,^. A procedure 
similar to the one* desc ribed above is followed except that , the grade- - 
equivalent scale i«' empirically anchored at two points instead of one. 
The fall-to-spring interval, however, is still arbyttrarily divided into 
equal intervals ^(the number of mpnths between test points) and the spring- 
to-fall ^int(/rval is likewise divided, again considering t^ie 3-morith 
summer to re^res^it 1 month of growth. This .procedure , while a little 
more soundly^ b^sed for the average student, is still dependent upon- the 
median student and reflects no empirical data for low-achieving students. 

• Despite these problems, the j^i^pular appeal of the month-f or-month 
standard is under^taildable . If one believes^ tliat this is the rate of 
achievement for the Average student and thus higher than that for the 
below-average student, it is reasonable to conclude that a program is 
effective if it produces month-f or-month rates of gain for .disadvantaged 
students, [ 

Ten Perce ntile Points 

^ 7 ~. / • 

The second standard that we apply Is an increase of 10 percentile 
points from ffret|st to posttest. The use of a percentile point increase 
as. a st^^anKls based on the assumption that a student is expected to 
maintain the sW/ percentile ranking from one test level to the next and 

8 



from one time to, the next. Thus, in the absence of ^an intervention, a 
student who scores at* the 20th percentile at the beginning of first grade 
would be expected henceforth t^ score at the 20th percentile. In other 
words, the test*' norms assume that relative rank among individuals is 
preserved* It should be noted that this standard is not dependent on 
time, as ratei» of jgaln are; that is, an increase of. 10 percentile points 
is considered significant whether it occurs over a 3-month period or a 
3<ryear period. _ ^ 

The choice of ten percentile points st^ms from the need to have a 
shift that is large enough to be educationally ,signif leant while mini- 
miziog the possibilities of chance fluctuation. Alttiough it \s impossible 
to determine precisely when a difference is large enough to have educa- 
tional meaning, evaluators such as the R!MC Research 'Corporation have 
applied a rough rule of thumb: the gain should equal or exceed one- 
thlrji of the standard deviation of the norm group. We roughly estimated 
the equivalent of one-third of afn average standard deviation by first 
translating one standard deviationfor each test and grade level into- ' 
percentile points. We then averaged\thes^ across tests and grades 
atjid arrived at 30 percentile points, one-third of which is 10 percentile 
points. 

Overall, th^ ten-percentile standard i» somewhat arbitrary and 
extremely stringent — one which to our knowledge has never been met. For 
example, the final analysis of the nationa^l Follow Through evaluation 
data uses a standard of 1/4 standard deviation, which i^ not achieved in 
a large majority of the comparisons made.' Nevertheless, for purposes of 



*The translation varies somewhat across the distribution of test scores. 

f^For example, at the 50th percentile, an increase of one-third of a 
standard devlation'on the CTBS" roughly translates into a 13-point 
percentile increase compared ^with an 11-point percentile in^crease at 
the 30th percentile. This would pose a serious problem if we were 
dealing with the entire range of test scores. Because oUr calculations 
were limited to the lower portion of the distribution (centering around 
the 20th pe^rcentile) , the problem is minimized, but not eliminated. 
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illustrating the Impact of -different standards currently in use, it 
^erves well. 

Eight Months Annual Gain 

The third standard we apply Is the achievement of 8 grade-equivalent 
months during a 12-month period. This standard, expressed in the language 
of test norms, is based on ^n expectation of 7 months annual gain for 
disadvantaged students. It differs from the month-f or-mpnth standard in 
that empirical data support this figure as an estimate of expected annual 
growth for disadvantaged students. One source for such support is the 
data collected by the states in evaluatinrg Title I programs. If one 
divides each, grade 'js mean pretest score by the' number of years the 
students have been in school, the average annual growth is approximately 
7 months across all grades (Thomas and Pelavin) . The pretest scores 
probably include some students who were previously in Title I, suggesting 
that the expectation, if biased, is an overestimation . Based on this 
expectation, we have chosen a 1^-month increa'se over expected achievement 
(that Is, a total of 8 months achievement), as our third standard for 
judging effectiveness. 

Section III describes *the type of data siought and obtained for our 
analyses. This is followed by a description of the analyses and the 
results. ' " ^ ' 



This 1-month increase is not related to the expected L-month summer 
gain for the 50th percentile student. We have arbitrarily defined the 
standard to be 1 month greater than the 7-month annual expectation for 
the disadvantaged student. Although the 8-month standard is an annual 
standard, we justify applying it' to the shorter Jall-to-sprlng period 
in light of our hypothesis that losses occur over the summer. 
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Ill DESCRIPTION OF SEARCH FOR DATA AND DATA SETS OBTAINED 



To be able to carry out comparisons between dif f erent ' time periods 
and to apply different standards, we required data with certain charac- 
teristics. Ideally, we wanted fall and spring standardized achievement 
test scores for individuals in raw score ||prm for several consecutive * 
years and several waves of students/ Consecutive years of fall and 
spring testing permit a comparison of evaluations based on a schools- 
year period 'as well as a 12-month period. Raw scores permit transforma- 
tions into grade-equivaTents and percentiles, thereby allowing application 
of the three aforementioned standards. 

We restricted our search to current programs' so that we could 
observe them in^operation. Since we limited ourselves to programs 
whose stated objective is to increase achievement as measured by. 
Standardized tests, we required some assurance that the operating program 
was in fact primarily academic. We wanted to eliminate the pi^ssibility 
that the data might be based on programs that, in fact, did not really !y 
exist. We did not, however, pursue the issue to the point of-investi- 
gating the extent to which the curricular content of the program matched 
the content otf the test. , 

ThQ remainder of this- section includes a brief review of previous 

f 

research on the ef f ectivenes;s of compensatory education programs. In 
addition, we describe our seai/ch for data and the data sets obtained 
for analysis. • ^ ^ 

Previous Research ' , 

Our review was, carried out with the idea investigating the 
-/sustained effectiveness of compensatory programs . Therefore, we^ concen- 
trated on locating research that included measures taken after, the 
students had completed a program. Our review of the presl^ool literature 
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drew heavily on four excellent and comprehensive reviews (Stearns, 1971; 
White et al., 1973; Bronf enbrenner , 1974; Goodson and Hess, 1976), All 
th^ reviews indicated that substantial evidence exists C^^show significant 
shorts-term effects as measured primarily by standardized intelligence 
tests given at the end of a program. The evidence for sustained effects, 
based on measures taken at vary/^g times after program participation, 
suggests that most cognitive gains made in preschool disappeal: by the 
second or third grade. Pai^ent-child intervention programs are^ a possible 
"^exception. While these conclusions from the preschool literature are 
not beyond question, .they at le^st represent a consensus of several 
.reviewers. No such consensus exists beyond preschool. 

For the early grades, Grades K-3, our review uncovered a considerable 
amount of research on short-term effectiveness (the references for these- 
studies are in Appendix A). However, we were able to find virtua^y no 
work on sustained effectiveness.-^ A study is currently under way that 

:a^ned effectiveness: Th^ Office of ^ 
ig Effects. of Compensaftory Education on • 
Basic Cognitive Skills." Preliminary results from this study are not 
expepted before 1979, and th^flnal results sever!al years later. 

In the remaining grades. Grades 4-12, -there has again been research 
on short-term effectiveness (alsjo^ referenced d-Xi Appendix A), but ' no wc5rk 
on sustained ef f ect ivenejgrs . This re'^earch is not as extensive, by grade, 
as the research done on preschbols or Grades K-3, probably because there 
are far fewer compensatory programs at grade levels beyond Grade 6. 

Given |:he paucity of studies beyond preschool with n^asures of 
sustained effects, we w^re unable to draw from our review any conclusions 
about sustained effectiveness of compensatory education programs lA the 
elementary and later grades. 



is designed to investigate suaf 
Education's "Study of Sustaii*^ 



Search for Data 

We limited the search for data to compensatory prograids beyond 
preschool with emphasis on th« later grades. Our search for adequate 
data included a thorough review of projects identified in previous 



12 



searches for "exemplary" programs, an examination cff ERIC and the Current 
index of Education Journals, and a phone -survey of large cit||^8. Addi- 
tionally, we investigated data collected ^s part -^f the evaluation of the 
Voucher Demonstration in the Alum Rock School District* 

We. devoted consiiierable resources to tracking down projects previously 
identifie<f as "exemplary" in USOE-sponsored re^ea^ch done by the American 
Institutes for Research and the RMC Research Corporation. Since this^ 
prior research had been concerned with the quality of data, we felt the 
exemplary programs were our most promising source of adequate data. -Of 
the over 40 projects reviewed, we found 15 that might have adequate data. 
Of these, ^ight were immediately eliminated, when it was found they nd 
longer existed. Six did not have data that wo#ld support reanalysis,' 
and one program had adequate data^ but obtaining it would l^ave been 
prohibitively expensive. - /j ' »■ ' 

We were*quite surprised that phis research did not result in the 
location of usable daf5(, and' that so few of the "exemplary" programs 
were Still in Existence. Because the results of this search were sur- 
prising,, we have recorded the process involved and the findings in ' ^ 

considerable detail in Appendix B. . - ^ i 

Through our searches of ERIC and the Current Index of Education 
Journals, we identified two compensatory programs that might have ade-: 
quate data. Although we reviewed a large numb er^ of studies, very few 
reported achievement test data. Most contained very general evaluation 
data such as teacher judgments. Of the two promising candidates, one 
was eliminated 'because the |:esting had not been systematic and the sample 
of program participants with the s^me tests for more than a year was 
extremely small. The second program was the Dlagnostic-Prescriptive- 
Indivldualized Primary Reading Program in Louisville, Kentucky. We 
contacted the county school district and Obtained permission to 
reanalyze their data. On receiving and attempting to analyze the data, 
hqwever, we discovered limitations that preclj^ded their use for this 
report. . 
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Through phone calls to the. 24 cities with the largest populations 
betv/een the ag^s of 18 and 35, we located six metropolitan districts 
with potentially usable d'ata from evaluations of compensatory education 

■programs. Of tjiese six districts, two had progr^ams whose data met most 
of our criteria: High Intensity Learning Centers in ofaha, Nebraska 
and the California State Demonstration Program in Mathematics in Long 

♦Beach, California. We therefore contacted each of the programs and 
obtained permission to reanalyze their data. 

In Long Beach, we were told about California State Demonstration 
Programs in other junior high schools. We contacted the^Demonstration 
Program in Reading in Santa Barbara, California, and ^btained ^permission 
to reanalyze their data. ^ 

Data from the evaluation of the Voucher Demonstration in the Alum 
Rock School District in San Jose, Calif ornia'met most of our criteria. 
In using thes^ data, we recognized^that increasin&^chievem^nt was only 
one of many goals of the program^- and perhaps not a primary goal. We' 
obtained permission for our reanalysis from the Rand Corporation, which 
Uad collected the data, and^from the National' Institute of Education, 
which sponsored the demonstration. . . ' 

We report on the reanalysis of data from a total of four compensatory 
education programs. The programs and the characteristics of the data 
are described below. ■ . 

Data Sets , * . . 

_ . ^ , I. ' ■ ^ ' " 

The four data sets subjected to reanalysis represent twp state- 
funded compensatory education programs in California, onte Ti'Me I program, 
and the Voucher Demonstration in Alum Roclc. A 'brief description of each 
program and the' characteristics of the data obtained -ar^ given below. 



*We woulcf have liked to' h^ve detailed information on summer school 
participation foF each ^program but were unable to «9btain it. . For 
three of the f9ur programs'^ (excluding Alum Rock' for which we have no 
information) ,• the program directors felt that very few student? 
attended, summer school programs but they did not have exact numbers 
nor individual data. - ^ 
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Appendix C contains isome notes on the process and problems involved in - 
obtaiiring and transforming the available data into a form amenable to 
reanalysis. ^ , 

• California State Demonstration Programs in In tensive Instruction 
in Reading and Mathematics 

In 1969, the California State Legislature made funds available for 
the implementation of demor0irat ion programs in reading afid mathematics ' 
at the^unlor high school level. The intent of the legislation was to 
provide instructional aid to all students ill about 20 junior high schools 
with high concentrations of educationally disadvantaged youth. The 
program began in Grade 7 in January 1970 and moved with the students tp 
Grade 8 irt 1970-71 and Grade «9 -in 1971-72. In 1972-73, the 3"-year cycle 
began again. Additionally, in som6 districts other compensatory funds 
were used to replicate the program ^in those grades not supported by the 
State. We obtained data from two such programs: a mathematics program 
in Long Beach in which district funds were used tp support the program 
in years not funded by the State, and a reading program in Santa Barbara, 
which did not have district-funded replications. 

De/monstration Program in Mathematics (DPM), L ong Beach, 
California 

Program Description — In Long Beach, the DPM served all students 
in two junior high schools, beginning in 1969-70 and 1971-72, respectively 
The assumption underlying th^ mathematics program is that . junior high 
school students can increase their competence in mathematics most effec- 
tively if they are given individualized instruction fitted to tbeir needs. 
The program^ s staff have prepared a large variety of materials geared 
to individualized instruction including study packets designed to teach 
750 behavioral objectives, criterion-referenced pretests and postfcests 
for various skills and' concepts, laboratory lessons, and review sheets. 



Initially, each student is administered a criter ion--ref erenced, 
test to determine where in a sequence the student should begin. The 
program in each classroom begins each day with a Quickie Quiz, which Is 
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a motivational technique for focusing the attention of the students. 
When the quiz is completed (3 to 5 minutes),^ one-fifth of the studehts go ^ 
to a mathematics laboratory. Thus each student spends one day each week 
in the laboratory instead .of the regular classroom. The laboratory 
lessons are designed to match the students* classroom work and' are 
presented under the direct supervision of the lab teacher ,and teacher' 
aides. At the end of the Quickie Quiz, students remaining in the class-- 
room complete a short drill session using xeview sjfeets, and. then proceed 
with their individual packets. This procedure is f (flowed throughout 
the school year in all grades. 

Data Descri ption — Students in the program were administered 
the mathematics portion of the Comprehensive Test of Basic Skills (CTBS) 
^nually in early to middle October and May. In the first year, 1969-70, 
the pretest was not given until January. All students received Form Q3 
of the CTBS in Grade 7 and Form R3 in Grade 8. In Grade 9, the level 
changed to R4. The tests were administered by counselors and members 
of the district evaluation staff and scored by the evaluation staff. 
We obtained data in raw scores for four cohorts of sti^dents. For two 
of the cohorts, there were data from attest given subsequent' to partici-- 
pation in the program (administer'ed as part of the district testing 
program) . 

Demonstration Program in Reading (DPR), Santa Barba ra, California 

Program "Description — The reading program in Santa Barbara began 
in the seventh grade during the 1969-70 school year and continued with 
this wave of students through ninth ^rade. The program is in fact two 
"separate programs: a developmental program for students considered to 



*For one group of students, Cohort A, one of three different levels of 
CTBS (R2, R3, or RA) was administered based on a student *s preceding 
spring score. Additionally, the first groups of students. Cohort 1, 
received Form Q3 in Grade 8 and Form QA in Grade 9. 
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be* average or above average and a remedial' and enrichment program for 

the other students. Because we were mainly interested in students 

considered to be educationally disadvantaged, we .concentrated on the 

remedial and enrichment component. The remedial /ind enrichment program 

> 

was developed on the belief that learning problems are a function of 
environmental, academic, and psychological factors, and that students 
learn in different ways. Therefore, in addition to an 'eclectic classroom 
approach, the program uses the services of ^ staff counselor, a nurse, 
psychologists and honf^ visitoi^s* 

' Students identified as having reading problems spent 45 minutes 
daily in the Reading Complex. Those identified as having severe problems 
may have spent two 45-minute periods in the Reading Complex. The periods 
of read'fng are primarily individualized and small group instruct^'ion. 
Students' needs are identified on the^'basis of a variety of tests as 
well as information from the counselor, psychologist, or others acquainted 
with the students. The classes are small, 10-15 students, with a teacher, 
a teaching ass'lstant, and- usually a student teacher or adult volunteer, 
who employ a variety of instructional approaches and materials . The 
curriculum stresses, through reading, concepts such as cause and effect. 



which are taughrt wh^n possible through problem-solving situations. 



taug^ 

Inductive reasoning, , and discovery. Also, when th^i?^' sch^ule permits^ 
students can attend the Reading^ Complex at any time in addition to 
their scheduled periods of participation. ' ^1 

Data De scription — We obt^ned data in raw scores forgone cohort 
of students in Santa Barbara — those starting the program In Grade 7 in 
1972-73. These stiidents were administered the reading/ portion of the 
Comprehensive Test of Basic Skill#^ (cift) in October and May of each of 
the three years. Form Q3 of the CTBS was given in Grade 7, Form Q3 or 
R3 in Grade 8, and Form R4 in Grade 9., The tests were administered and 
scored by the program's staff. 
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High Intensity Learning Centers (HILINC) , Omaha, Nebraska ' 

Program Des cription — In 1971'^72, Omajia adopted High Intensity 
Learning Centers (HILINC), a program developed- by Random House Publishers. 
HILINC s purpose Is to Improve reading comprehension test scores of Title I 
students. The tirogram serves approximately 3,500 students annually in 
Grades 3--12 in Title I schools. One or more High Intensity Learning 
Centers is at each participating school. Each center is staffed by a 
teacher and one teacher aide. Participating students, selected on the' 
basis of previous test scores, spend 1 hour daily in th^ program in 
addition to their ' regular readfng class. Initially, each student is 
diagnosed on^ttie- basis of an instructional objectives test. Specific 
materials and activities are then prescribed. These materials are 
intended to be self-directing and self-correcting, and are sequenced so 
that pupils need a minimum of teacher direction. While the materials 
used initially were those of the publisher, over the 'last 3 years the 
original program has been almost entirely replaced by materials written 
by the reading consultants and teachers. 

Data Description — Omaha evaluates its Title I pij-ogram on the basis 
of fall (early October) and spring (mid-May) administrations of the 
Reading Comprehension Subtest of the Gates-MacGinitie Reading Test. The 
level of the test is determined by a student's instructional level. 
Thus, for a given grade^ students mdy receive any of several levels of^ 
the test. The tests were administered and scored by teachers in the 
program. We obtained scores in grade-equivalents for students in Grades * 
3-8 for the school years 1971-72 through 1974-75. 

The Voucher Demonstration in the Alum Rock School District ^ 
Sap Jose, CajLifornla 

Program Desdrlptlon- -In 1972 the Federal Office of Economic Oppor- 
tunity (OEO) authorized a Voucher Demonstration in the Alum Rock School 
District. This demo!tstrat ion included 6*of the- district's 24 schools 
serving students in Grades K through 8. Each school was required- to 
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provide at laast two "mini-school^" (program options), with the district 
supplying the basic 'voucher from its current income and OEO providing 
compensatory vouchers for children who qualified for the Federal free 
lunch program. In 1973-74, the demonstration expanded to 13 schools 
with about 9,000 students and 45. "mini-schools ," and the National Institute 
of Education took over sponsorship of the demonstration. 

\ The Alum Rock Voucher Demonstration does not have one "program" 

in the sense of an identifiable classroom model with a specific educa- 
tional goal. Instead, it reflects. a large number of goals that vary 
somewhat from year to year. In this way it differs considerably from 
the other programs included in this study. There seems to be general 
agreement that the original intent of the demonstration has not been 
realized. The primary purpose of the program now seems to be to decen- 
tralize school-district authority and to provide parents with some freedom 
in the selection of a school program for their children^ Given this 
purpose, it is certainly not obvious that standardized achievement tests 
should be the primary outcome measure, although there is clearly a 
consensus that one of the many goals of the demonstration is to increasie 
cognitive .achievement . This concern is discussed more fully with the 
presentation of the analysis results in Section IV. 

Data Description — The Rand Corporation directed the testing program, 
which consisted of the adminlstratioti of the Metropolitan Achievement 
Tests (MAT) in the fall and spr^.ng during the years 1972-73 through 1974- 
75. The tests were givei^ in November and May of the first year and s 
October and April of the next ^2 years to' students in Grades 1-8. The 
tests were administered by a variety of personnel including classroom 
teachers, members of the district's evaluation staff, and substitute 
teachers. The tests were scored under the auspices of the Rand Corpora- 
tion. We had access to raw score data for all students tested. The 
data were complicated by the fact that there was no consistent patterp 
in the selection of alternative forms and levels of the tests. As a 
result, for each test point, a variety of levels and forms pf the test 




were administered to students In a given grade, so that a particular 
student of,ten did not receive the same level of the test more than twice. 
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IV ANALYSIS OF PROGRAM EFFECTIVENESS ' 

r 

The data are presented separately by program and discussed In the^ 
following way^ First we present the three means for the sample of stu- 
dents who were tested three times. From these means , the achievement 
gain and the rate of achievement are calculated for both the 9-month, 
school-year period and the 12-month, fall-to-fall period. The achloye- 



ment gain and the rate of ^achievement for the school-year period are then 
compared w:^th the gain and rate for the 12-month period. We then apply 
the three standards— -a 10-polnt percentile Increase, a gain of 8 grade- 
equivalent months, and an achievement rate of^l grade-equivalent month 
pet month — to the results for each time period. This allows us to com- 
pare the extent to which conclusions about program effectiveness vary 
both under different time periods and with the application of different 
standards. 

0ur discussion Is extended to 2 years of a program by Vising samples 
of students who have had five tests administered to them: fall and 
spring of 2 consecutive years and fall of a third year. We present 
these five means with the achievement gains based on t?hree different 
time periods: the two fall-to-spring periods, fall of the first ye^r ^ 
to spring of the second year and fall of the first year to fall of the 
third year. To demonstrate the extent to which the Inclusion of the 
summer months affects an evaluation, these time periods reflect the exr 
elusion of both summer Intervals, the Inclusion of the Intervening sutiSh 
mer, and the Inclusion of both summers, respectively. We then consider 
these findings In the context of the three standards described above. 
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Because the standards that we apply are in terms/ of grade equiva- 
lents and percentiles, we report only these metriqa^ in the text.* For 
data ^ets that contained standard scores as well-/we report these scores 
and th^ir standard deviations in the appendices.! To simplify the text 
further, we present in the tables summary figures averaged across cohorts 
of students. In general, for ^11 the data sets the pattern of the means 
for each cohort follows the pattern of the means averaged across cohorts. 
.The data, broken down by cohort, are also presented in the appendices. 
References to the corresponding appendix tables appear in the text for 
each program. 

P PM In Long Beach 

From the Long Beach DPM we obtained data for four groups of stu- 
dents: students who began Grade 7 in 1969-70, 1970-71, 1971-72 and. 
1972-73. Table IV-1 presents data by grade level for all students with 
three test points (fall and spring of one year and fall of the next)*^ 
averaged across four groups.. The first three columns span a 12-"month 
period and contain the grade-equivalent and the percentile scores asso- 
ciated with each standard score mean for each test administered for 
Grades 7 and 8. These statistics, as well as the standard score means 
and standard deviations, are presented separately by grade, school, and 
cohort in Appendix D. We are primarily interested* in comparing the 
achievement over the traditional fall-to-spring evaluation period with 



With the exception of the Omaha program, wtv^ch reported only grade- 
equivalent scores, the means were always calculated in standard scores 
and then translated into grade-equivalents. This avoids the problems 
associated with averaging grade-equivalents. For all the data sets, 
we compared calculations based on means and medians^ and found no dif*-' 
ference in the resulting patterns. 

^To determine if our samples are representative of all students in the 
program, we have compared our samples to all students tested at a given 
point. We have found no systematic differences between the means and 
standard deviations of our samples and the larger, cross-gectional 
groups. In general, where there are differences they tend to favor 
the longitudinal groups, which is not surprising since they probably 
represent^ a more stable group. The cross-sectional data are presented 
in Appendix D. 

• * • • 
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Table IV-1 

LONG BBACH DPM CTBS MATHEMATICS hfEANS IN GRADE-EQUIVALENTS 
-AND PERCENTILES AND GAINS FOR TWO TIME PERIODS 



Means 



Gains 



Sample 



Grade 7 (n-780) 
• Grade-equivalent 
Percentile 

Grade 8 (n-A68) 
Grade-equivalent 
Pereentlle 



I 

Fall 



5.5 

23 



6.4 
26 



II 
Spring 



7. A 
45 



7.9 

38 



III 
Fall 



6.6 

28- 



7.8 

30 



IV V 
Fall to. Sprlnp, Fall" to Fall 



1.9 

22 



1.5 

12 



1.1 

5 



1.4 
4 



.the achievement over the 12-inonth, fall-to-fall period. Comparing the 
grade-equivalent and percentile .means In Columns II and III, the second 
fall score is lower than the spring score for , both grades. Therefore, 
the fall- to-f all estimates of achievement (Column V) are smaller than 
the fall-to-spring estimates (Column IV), The small difference in grade 
equivalents for Grade 8 reflects th? small difference in the means. 
Since the level of the test changed between the spring ofTGrade 8 -and ? 
the fall of Grade 9, the smaller sutraner loss for the Grade 8 samples 
may be a function of the level change.. Since the test level change is 
completely confounded with program ^^3icipation in Grade 8, it is im- 
possible to be certain of the cause. 

We now consider the impact of these summer losses on conclusions 
about program effectiveness as judged by the .three standards described 
above. First, we inspect shifts" in percentile scores under the assump- 
tion that they would remain the same, on the average, in the absence of 
a program, impact. We then compare increases in percentile to our most 
stringent standard, that of a 10-point increase for the two time periods, 
fall to spring and fall to fall. Looking at the percentile differences 
for the fall-to-spring period in Column IV, we see that there is a sub- 
stantial percentile increase for both grades: 22 and 12 percentile * 
points. Both of these increases exceed the 10-point standard. However, 
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if one looks at the percentile changes for the fall-to-fall perio3^ a 
Column V, a very different picture emerges. 'Here the percentile in- 
creases are only 5 and 4 points. For this time period, neither grade 
reaches the standard of a 10-point increase. Hence, while the program 
would be judged quite effective for a f all^to-spring period, it would 
not be judged so for a fall-to-fall period. 

Our second standard is a gain of 8 grade-equivalent months (0.8 
grade-equivalent years) per year. If we look at the grade-equivalent 
gains in Column IV for the fall-to-spring period (less than a year), the 
program looks extremely- effective. The Grade 7 g^^in Is 19 months and the 
Grade 8 gain is L5 months. When the summer is included in the period 
over which the gain is measured, however, the gains are much smaller 
(see Column V). Nevertheless, while the gains are smaller for the fall- 
to-fali period (11 and 14 months), both grades still exceed the standard 
of an 8-month gain- per year; Hence, the program would still be con- 
sidered effective. 

The third standard is a gain of 1 grade-equivalent month per month. 
Table IV-2 gives the average monthly grade-equivalent rate for the fall- 
to-spKing (Column I) and fall-to-fall (Columnll) periods. These rates 
are calculated by dividing the fall-to-spring and fall-to-fall gains 
from the totals in. T^fele D-3* by 7 and 10 respectively .+ 

Comparing Column r with Column II in Table 1^-2, we see that again 
the rates are substantially smaller for the fall-to-fall period. If the 
program is judged on the basis of the f all-to-spr ing rates, it is quite 
effective, with monthly rates of 2.8 and 2.1 months per month. However, 
these rates diminish considerably when calculated over the fall-to-fall 



"^The appendix tables provide the rates averaged across cohorts. There- 
fore the rates are slightly different than those calculated directly 
from Table IV-1 due to rounding error. 

+The divisor for the fall-to-spring period is 7 since the interval be- 
tween the fall and spring adminl^strations of the CTB6 is 7 mouths. Th^ 
divisor for the annual rate is 10 since, the grade-equivalent year. con- 
tains 10 grade-equivalent months, 9 for the school year and one for 
the summer. " . - , 
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Table IV-2 

LONG BEACH DPM CTBS MATHEMATICS MONTHLY ACHIEVEMENT RATES 
IN GRAjDE-EQUIVALENT MONTHS FOR^TWO TIME PERIODS 

MontMy Achievement Rates 
.1 II 
Sample Fall to Spring Fall to Fall 

Grade 7 .(n=780) 2.8 lj2 

Grade 8 (n^468) 2.1 1.3 



period. For this time period they are 1.2 and 1.3 months p'er month. 
Nevertheless, even for the fall-to-falx, period, the program overall is 
still effective, with both rates in excess of 1 month per month. 

No matter which standard is applied, we argue that fall to fall 
is the appropriate period of time for judging program effectiveness.. If 
an evaluation is based on a traditional fall-to-spring period, the re- 
sults will not reflfect the extent to which gains have lasted, at leieist 
until the beginning of the next school year. The Long Beach data illus- 
trate that for 1 year, the fall-to-fall gains are consistently smaller 
than fall-to-spring gains.- However, the gains are sufficiently 
large during the school year that, in spite of large summer losses, the 
program is judged effective under two of the three standards of effec- 
tiveness over the 12-month, fall-to-fall period. 

We now extend our analysis to judgements of* 2 years of the program 
with a sample of students who were tested^ f Ive times : fall and spring 
of Grades 7 and 8 and fall of Grade 9. For each of the five test admin- 
istrations Table IV-3 presents the grade-equivalent mean and the percent- 
ile associated with. each standard score mean. Appendix D presents these 
data, as well as the standard s'core means and standard ' deviations , selpa- 
rately by cohort and school. 

This 2-year sample reflects the same' pattern as the two 1-year 
samples described above. There are. losses over both summer intervals 
(Column. Ill minus Column II and Qolumn V minus Column IV). Again, it 
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Table IV-3 

LONG BEACH DPM CTBS MATHEMATICS MEANS IN GRADE 
EQUIVALENTS AND PERCENTILES FOR TWO YEARS 



Means 



Sample 



Grades 7-8 (n=378) 

Grad e- equi va 1 en t 
Percentile- 



Grade 7 
Fall 



5.5 

23 



II 

Grade 7 
Spring 



7.3 

43 



III 
Grade 8 
Fall 



6.6 

28 



IV 
Grade 8 
Spring 



8.1 
40 



V 

Grade 9 
Fall 



7.9 
31 



should be noted that the difference over the second summer reflects a 
change in test level, which may or may not explain the smaller loss. 

Since there are losses over both v^ummers, 2-year estimates of 

achievement will be largest if neither summer is included; that is, if^ 

2-year achievement is measured as the sum of two fall-to-spring gains. 

* 

This time period yields a .gain of 1.8 years (Column IX minus Column I) 
plus 1.5 years (Column IV minus Column III) which is a gain of 3.3 grade- 
equivalent years for the 2 years. If the estimate of 2-year achievement 
includes the intervening summer, the estimate of achievement is lowered 
to 2.6 grade-equivalent years (Column IV minus Column I). Finally, if 
both summers are included, the achievenient estimate is even smaller — ^r4 
grade-equiv^alent years (Column V minus Column I). Similarly, in the 
percentile metric, the sum of the two fall-to-spring gains is 32 per- * 
centile* points. Inclusion of the intervening summer reduces the gain 
to 17 percentile points, and the inclusion of both surayiers lowers the 
gain to 8 percentile points. * • . 

In comparing the differences und^ the three time periods to the 
10-point percentile standard, it is obvious that the. 32 percentile 

r . ' 



""if the Grade 8 f^ll score Reflects any part of the Impact of the Grade 
7 programs, creating the two fall-to-spring gains s^eparately is mis- 
leading. Logically, the Grade 7 fall store should serve as the ex- 
pected p'ercentile throughout the program. 
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point gain calculated by summing the two fall-to-spring gains greatly 
exceeds the 10-point standard. The percentile shift is smaller (17 
points) when measuring from the fall of Grade 7 to the springrof Grade 8, 
but still large enough to meet the standard for program effectiveness. 
However, when both summers are included in the evaluation by measuring 
from the fall of Grade 7 to the fall of Grade 9, the percentile increase 
of 8 points no longer reaclTfes the standard. Hence, under the time period 
measuring sustained effects, the program would not be judged effective. 

« We now compare the grade-equivalent gains to the standard of an 8 
grade-equivalent month gain per year in order to evaluate the success of 
th^ program. This means that the effectiveness of a 2-year program 
should be judged by comparing the 24-month gain (fall of Grade 7 to fall 
\ of Grade 9) to 1.6 grade-equivalent years (a gain of 0.8 year or 8 months 
for^each year). For all three time periods the program is effectives- 
using this standard. While the inclusion of both summers gives the 
smallest gains , ^the fall of Grade 7 to the fall of Grade 9 still reflects 
a gain of 2.4 grade-equivalent years, which exceeds the standard of 1.6 
grade-equivalent years. a , - 

Turning ^o Table IV-4, we now compare the 2-year rates of growth in 
grade-equivalent months to the standard of a month-f or-month gain. The 
first two columns present the monthly rates based on the two fall-to- 
spring intervals, these rates are 2.8 and 2.0 months per month, respec- 
tfvely.* If the program were judged on this basis it would be considered 
effective over a 2-year period by virtue of greatjy exceeding the stand- 
ard in both years. If the program were judged on a time frame including 
the intervening summer, the rate 6f 1.6 months per month still exceeds 
the month-f or-month standard. Finally , judged on the full 2-ye^r time 
period (Column IV), the rate of 1.3 months per month is even smaller 
but still exceeds the standard. 



These rates are based on the totals in Appendix Table D-5, which are ' 
the rates averaged across cohorts. Therefore, the rates are slightly 
different than they would 'have been if calculated directly from Table 
IV-3, due to rounding error. 
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Table IV-4 



LONG BEACH DPM CTBS MATHEMATICS MONTHLY ACHIEVEMENT RATES 
IN GRADE- EQUIVALENT MONTHS FOR THREE TIME PERIODS 

Monthly Achievement Rates 

1 II III IV 
Grade 7 Grade 8 ' Grade 7 Fall Grade 7 Fall 
Fall to Fall to fo Grade 8 to Grade 9 
Sample Spring Sp r ing Spri ng Fall ' 

Grades 7-8 

(n=378) 2.8 2.6 , 1.6 1.3 

In summary, the Long Beach data illustrate that es{:imates of achieve- 
ment and effectiveness can, vary tremendously when different time frelmes 
are used in both 1-year and 2-"7ear evaluations; While the Long Beach 
program continues to look ,ef feet ive' under allv^time peripds for the two 
grade-equivalent standards, it is important to keep in mind. that the in- 
clusion of the summer months does redupe the size of the achievement 
gains and, in the case of the lO-poirft percentile standard, changes the 
conclusions reached. ^ 

DPR in Santa Barbara ' 

For the Santa Barbar^ reading program, we have data f or ^nly on^ 
cohort of students, those who 'entered^ Grade 7 in 1972-^3. Columns I, 
II, and III in Table.IV-5 t:ontain the grade-equivalent and the*" percentile^ 
associated with each mean'foi? three test administrations for all students 
tested in fall and spring of 'one year and fall of the ne^t..^ The stand- 
ard score means' and standard deviations are pfes'ented in- Appendix E. 
Columns IV and V give the gains from fall to sparing and fall to f&ll, 
respectively. For both graces, there is a loss of aqhiaveirtent during 
the summer', this summer loss is reflected in the coii/parison between the 

y - ■ ' . ^ 



*We 4phen compared the means, and standard deviations of these samples to 
all students tested at each test point and found no differences. See 
Appendix E/ ' ' • 
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Table IV- 5 

SANTA BARBARA DPR CTBS READING MEANS IN GRADE- EQUIVALENTS 
AND PERCENTILES AND GAINS FOR TWO TIME PERIODS 



Means Gains 

I II III IV V 

Sample Fall Spring Fall Fall to Spring Fall to Fall 



Grade 7 (n«102) 

Grade-equivalent 4.3 5.6 5.4 1.3 1.1 

Percentile .12 20 16 ' 8 . A 

Grade 8 (n=«107) 

Grade-equival^tiif 5.5 6.5 6.2 1.0 0.7 

Percentile 16 23 16 7 0 



fall-to-spring and fall-to-fal]^ gains. The fall-to-fall estimate of 
achievement is smaller than the^f all-to-spring estimate by 2 grade- 
equivalent months in Grade 7 and 3 in Grade 8. This difference is also 
reflected in percentile shifts, where the gains are 8 and 7 percentile 
points for the two grades as measured from f'all-to-spring, brut only 4 ^ 
and 0 points for the two grades when measured from fall-to-fall. 

'x A compa^rison of these percentile shifts to the 10-point standard 
shows that th-e program does not meet the standard under either time 
period. However, a comparison of the gra^-equivalent gains to the 
standard of an 8-month gain per year shows that the program is effective 
in both grades from fall to spring. During the 12-month period, the 
program its effective in Grade 7 (a fall-to-fall gain of 1.1 years br 11 
months) but not effective in Grade 8 (a. fall-to-fall gain of 0.7 year 
or 7 months) . ^ 

Table IV-6 presents the monthly rates for the two time periods. 
Comparing these with the month-f or*-month standard, we see that for both 
grades the fall-to-spring rates exc-eed the standard (1.9 and 1.4). How- 
ever, during the fall7to--f all period, the incorporation of the sumn^er 

into the estimate lowers the rates to l.X and 0.7 month per month — only 

# 

the Grade 7 program is effective under the month-to-month standard. 
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Table IV- 6 ,»• 

i 

SANTA BARBARA DPR CTBS READING\; MONTHLY ACHIEVEMENT RATES 
'in GRADE- equivalent MONTHS OVER TWO TIME PERIODS 



Monthly, Achievement Rates 

I II 
Sample . Fall to Spring Fall to Fall 

\ 

Grade 7 (n=102) 1.9 1.1 

Grade 8 (n=107) l.A - 0.7 

Table IV-7 extends the data to 2 years of the program with means 
for students jfjith five consecutive test points (fall and spring of Grades 
7 and 8 and fall of Grade 9), Again, there is a loss during both summers 
1 grade-equi^falent month or 4 percentile points over the first summer 
(Column III minus Column II) and 3 grade-equivalent months or 6 percent- 
ile, points over the second summer (Column V minus Column JV) . Conse- 
quently, the inclusion of each summer in the evaluation tiiKe period 
reduces the size of the achievement gain. 

* We first compare the changes in percentile scores under the three 
periods to the standard of a gain of 10 percentile points. The sum of 
the two fall-to-spring gains is 14 percentile points, which clearly 

Table IV-7 

SANTA BARBARA DPR CT^fe READING MEANS IN GRADE- EQUIVALENTS 
AND PERCENTILES FOR TWO YEARS 



Means 



I II, III, IV V 

Grade 7 ^ Grade 7 Grade 8 Grade 8 Grade 9 



Sample " ^gli ' Spring fall ' Spring Falj 



Grade 7-8 (n=99) 
Ji^ 



Grade-equivalents- *4.3 5.6 5.5 6.5 6.2 

Percentile ' ^ 12 20 16 ' 22 16 




3,9 



exceeds the lO-percentlle point stan'dard.* Using only the fall-to-sprlng 
^galns the program would be judged effective. ^ 

yflie Increase from the fall of Grade 7 to the spring of Grade^S is 
10 Wrcentile points. However, if the program is judged on the basis of^ 
sustained gains over, both summers, and measured from the fall of Grade 7 
to the fall of Grade 9, the increase in percentiles is only 4 pcints. 
During this time period, the program would not be Judged effective. , 

We next compare the grade equivalent gains during the three time 
periods to the standard of an 8-month gain during each year or a 16-month 
gain during 2 years. For all three time period^, the program Is judged 
effective when using this standard. The sum of the two fall to spring 
gains is 23 months; the gain from the first fall to the second spring is 
22 months; and the gain from the first to third fall is 19 months. Each 
gain- i's greater than the 16-month standard. " - 

Table lV-8 contains the monthly rates in grade-equivalents for the 
three 2-year time periods. Under all three time periods the program is 
judged effective when compared with the standard of a month-f or-month 
gain. However, this rate is considerably smaller (1.0) when measured 
over the full two calendar years than when measured, by excluding one \ 
summer (1.3) or both summers (1.9 and l.A for the two school years). 



V. 



Table IV-8 

SANTA BARBARA DPR CTlfc READING MONTHLT ACHIEVEMENT RATES IN 
GRADE- EQUIVALENT MONTHS FOR THREE TIME PERIODS 



Sample 



Grades 7-8 (n=99) 



Grade 7 
Fall to 

Spring^.^ 

1.9 



Monthly Ach ievem e ht Rates 



IIv' 

Grade 8 
Fall to 
Spring 

1.4 



JII 

Grade 7 Fall 
to Grade 8 
Spring 

1.3 



IV 

£rade 7 Fall 
,to Grade 9 
Fall 

1.0 



If the Grade 8 fall «eore reflects any part of the* impact of the Gi'ade 
7 programs, creating the two fall-to-spring gains separately is mislead- 
ing. Logljcall^, the Grade 7 fall score should serve as the expected 
percentile throughout the program. 
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These findings again illustrate that results vary over different 
periods of tim^, and that such differences can affect conclusions about, 
program effectiveness. ~- — 

Omaha HILINC 

From the ^HILINC program^ in Omaha, we obtained data for six different 
cohorts of students in both public and nonpublic schools spanning Grades 
3-8 for a 4-year period. We present „only the d<ata frqm^ students in 
public schools averaged across cohorts. Data ^f or publj^c^chgol students 
are presented l)y cohort in Appendix F. Data for students in nonpublic 
schools are presented in Appendix G. Analyses were performed oi^tly^n 
/those grades and cohorts for which there were at least 20 students for 
whom we had received data. Since Omaha record^ test results only in 
grade-equivalents, our analyses were ^festricted to this metric. 

Table IV-9 contains the grade-equivalent means for all students 
with at least three- test points (fall and spring o'f one year and the 
fall of the next year).* We 'first compare tl^e means for the spring 

Table TV-9 

OMAHA HILINC GATES-MacGINITIE READING MEANS IN GRADE- 
EQUIVALENTS AND GAINS FOR TWO TIME PERIODS 







Means' 




Ga ins 




Sample 


Fall 


^ II 
% Spring 


III 

Fall , 


IV 

Fall to Spring 


- V 

Fall to Fall 


Grade 3 (n=272) 


2. 2 




2.8 


1.1 


0.6 


Grade 4 (n=931) 


2.6 




3.2 


1.0 


' 0.6 


Grade 5 (n=980) 


3.3 


4.3^ 




i:o 


0.7 


Grade 6 (n=316) 


^ 3.8 


4.8 


4.4 


1.0 


0.6 


Grade 7 (n=128) 


4.3 


5.2 


4.9 


0.9 


0.6 



^The corresponding data for all stiitients te'sted at each -point are pre- 
sented in Appendix F. While^ the cmoss-sec tional means are consistently 
higher than the longitudinal samples, the differences are extremely 
small. ♦ 



32 



41 



(Column II) with the means for the following fall (Column III). For all 
five grades, the fall means are lower than the means of the previous 
-springs Consequently , for- all grades achievement as measured from fall 
to fall is smaller than achievement from fall to spring. The differences 
between the estimates for the two periods of time (Column V minus Column 
IV) range from 3 to 5 grade-equivalent months. 

Since percentiles are not available, we cannot apply the percentile 
standard; therefore, we turn to the two grades-equivalent standards for 
assessing program success. Using the standard of an 8-month gain, all 
the grades exceed the standard during the school year. However, for all 
grades the inclusion of the summer loss reduces this gain to less than 
8 months. Thus, in every grade, the program would be considered effec-- 
tive if judged from fall to spring, but failing if judged from fall to 
fall. , ^ . 

Table IV-10 translates the fall-to-spring and fall-to-fall achieve- 
ment into monthly rates by dividing the achievement by 7 months (the 
number of months between the test administrations) and 10 months, re- 
spectively. Comparing these rates to the month-f or-month standar4, we 
see that for all grades the monthly rate as calculated from fall-to- 
spring exceeds the standard. These achievement rates range from a low 
of 1.3 to a high of 1.6 months jDer month. But if we judge the program 

Table IV-10 

OMAHA HILINC GATES-MacGINITIE READING MONTHLY ACHIEVEMENT 
RATES IN GRADE-EQUIVALENT MONTHS, FOR TWO TIME PERIODS 

Monthly Achievement Rates 



I II 

Sample Fall to Spring Fall to Fall 

Grade 3 Cn=l272) ' 1.6 - 0.6 

Grade 4 (n=931) 1.4 0.6 

Grade 5 (n=980) 1.4 0.7 

^ade 6 (n=316) 1.4 0.6 

Grade 7 (n=128) 1.3 0.6 
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on the basis of rates calculated for the 12-month year, the rat^s for 
all of. the grades are below the standard. These achievement rates range 
from 0.6 to 0,7 month per month. 

Table IV-ll presents ,the grade-equivalent means for those students 
tested at least five tim'es (fall and spring of' two successive years and 
fall of the next). Thede means are presented by grade range. The^'data 
are presented by cohort in Appendix F. All the grades show losses for 
both summers, ranging from 2 to 6 grade-equivalent months for the first 
sumner and 4 to 6 months for the second s'ummer. Tbrfl^, for all samples, 
the inclusion of the first summer in estimating achievement (Column IV 
minus Column^ I) will reduce the estimate from that based on the two 
school years. And the inclusion of both summers ^Column V minus Column 
I) reduces the estimate of achievement still further^ 

We now compare the gains over the three time periods to the 8-month 
standard. Since we are viewing 2 years of the program, the standard for 
effectiveness is a gain of 1.6 years. The sum of the gains for both 
years based on the two fall-to-spring periods (Column II minus Column I, 
and Column IV minus Column III) exceeds 1.6 yea|?s in all samples • The 
sums range from 1.7 years to 2.1 years. Therefoi*e, the program would be 
judged^ef f ective. When the gains from the first fall to the second 
spring (Column IV minus Column I) are used, only two of the samples 
reach the 1.6-year standard (Grades 4-5 and 5-6). The other two samples 

Table IV-11 

OMAIL^HILING GATES-MacGINITIE READING MEANS IN 
GRADE-EQUIVALENTS FOR TWO YEARS 









Means 






Sample 


I 

Fall 


II - 
Spring 


III 
Fall 


IV 
Spring 


V 

Fall 


Gra(5es 3-4 (n=87) 


2.6 


.3.6 


3.0 


4.1 


3.5 


Grades 4-5 (n=324) 


2.7 


3.5 


3.1 


4.3 


3.9 


Grades 5-6 (n=130) 


3.2 


4.1 


3.9 


4.8 


4.4 ■ 


Grades 6-7 (n=45) 


4.1 


4.9 


4.5 


5.4 


5.0 
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are close, having gains of 1.5 and 1.3 years, but would not.be judged 
effective by the 1.6-year standard. If one includes both sununer inter- 
vals in order to reflect sustained achievement, the gains (Column V minus 
Column I) range from 0.9 to 1.2 years. Under this time period, none of 
the samples reaches the 1.6-year standard. 

To judge the program against the standard of month-f or-month achieve- 
ment, we present the achievement rates for the three time periods in 
T^ble IV-12. If we compare the two fall-to-spring rates (Columns I and 
II) with the standard, we see that in all cases the school-year rates 
exceed the standard for both years. In fact, most of the rates are sub- 
stantially greater than the mqnth-f or-month standard. If we include the 
intervening summer in estimating the achievement rate (Column III), none 
of the samples reaches the standard. If we now include both summers in 
order to capture the extent to which achievement is sustained we find , 
another substantial drop (Column IV). The rates based on the peri'od from 
the first to the third fall range from 0.5 to 0.6 month per month. 

These findings provide a dramatic illustration of how conclusions 
about program effectiveness change when the evaluation time period in- 
cludes the summer months. This program is consistently effective during 
the school year but, because of large summer losses* cannot be judged 
effective for longer periods of time. 

Table IV-12 

OMAHA HILINC €ATES-MacGINITIE READING MEANS IN RATES IN 
GRADE-EQUIVALENT MONTHS FOR THREE TIME PERIODS 

. jtonthjj Achievement Rates 



Sample 



Grades 3-4 (n=87) 
Grades 4-5 (n=324) 
Grades 5-6 (n=l30) 
Grades 6-7 (n=45) 



I 


II 


III 


IV 


Fall to 


Fall to 


Fall 1 


Fall 1 


Spring 


Spting 


to 


to 


Year 1 


Year 2 


Spring 2 


Fall 3 


1.4 


1.6 


0.9 


0.5 


l-.J. 


1.7 


0.9 


0.6 


1.3 


1.3 


0.9 


0.6 


1.1 


1.3 


» 0.8 


0.5 
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Aluy Rock Voucher Demonstration 

For Alum Rock, we obtained 3 years of data including six cohorts of 
students in Grades 1 through 7. The re?ider should be reminded before 
inspecting the results that this is the one program that is not specifi- 
cally a reading or mathematics program intended to increase scores on 
standardized tests. Therefore, although the numbers are interpreted in 
the context of program effectiveness, c^oncludions should be drawn with 
caution. 

Ta|>14 IV-13 contains data for all students %.th three test points 
by grade. Columns I, II, and III contain the grade-equivalent score 
Associated with the mean for: each test/ administration. These data along 
with the standard score means and standard deviations are presented in 
Appendix H. Columns IV and V coqtain the differences in grade-equivalent 
means for the f^-to-spring and fall-to-fall periods, respectively. For 
all grades, the means are based on at least two different levels of the 
Metropolitan Achievement Test.* Therefore, the interpretation of any 
one ^lean presumes the adequacy of the, standard score scale and grade- 
equivalent scale across test levels.' 

The most striking feature of Table IV-13 is the remarkable similar- 
ity between the spring and subsequent fall scof-es, and hence between the 
fail-tO-sprMg and f all-to-f alJr^chievement ; Across grades, the largest- 
difference in gains for the two time periods is 2 motiths for Grade 7. 

A possible explanation for this finding is that the Voucher Demon- - 
stration is more an organizational scheme for schools than a program 
aimed specifically at reading instruction. Therefore, these scores might 
present a picture of untreated disadvantaged students. Without other 
d^ta on untreated .students, it is impossible to draw- this conclusion 



I 

^Because the means include scores from out-of- level tedts (levels not 
normed for that grade), percentile scores- are inappropriate and there- 
fore not included. 

+Two Studies of the MAT standard score scale have recently been* complete^ 
(Barker and Pelavin; Pelavin andi barker). Both studies indicate thafe ^ 
the §cale may contain biases, ^ 
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Table IV-13 

ALUMBOCK VOUCHER DEMONSTRATION MAT READING MEANS IN 
ORADE-^QUIVALENT AND GAINS FOR TWO TIME PERIODS 



. Means. Gains 



Sample 


I 

Fall 


II 

SpfinR 


III 
. Fall 


IV 
Fall to 
Sprinjj 


V 

Fall to 
Fall * 


Grade 1 (n»665) 


1.3 


* 1.7 


1.7 


0.4 


0.4 


Grade 2 (n«582) 


1.8 


2.4 


2.5 


0.6 


0.7. 


Grade 31 (n=781) 


2.5 ' 


3.1 


3.1 


0.6 


0.6 


Grade V'^(n=832) 


. 2.9 


3.5 


3.5, 


0.6 


0.6 


Grade 5 (n=842) 


3.6 


4.2 


4.2 


o\6 


0.6 


Grade 6 (n=728) 


.4.3 


4.8 


4.-9 


0..5 


0.6 • 


Grade 7 (n=813) 


5.3 


'6.0 


6.2 


0^.7 


0.9, 



with confidence. It is interesting to note, however, that these data 
reflect much smaller school year gains and smaller relative summer losses 
than those found in the three programs investigated above". 

We now compare the gains and rates over two time periods to the 
grade-equivalent standards. Neither the fall-to-spring nor the fall-to- 
fall achievement gains meet the standard of an 8-month gain except for 
Grade 7 fall to fall. All other gains fcCt both the fall-to-spring period 
and the fall-to-fall period range from 0.4 to 0.7 month. 

Table IV»-14 contains- the monthly achievement rates for the samples 
with three tests. 'These are calculated by dividing the achievement by 
6 and 10 months respectively (the number of 'months between the test ad- 
ministrations). A comparison of the monthly achievement rates i?Ver the 
two time periods to the mbnth-f or-month st«^dard makes tl^ differences 
over the two time periods more pronounced, .the seven grades, four 



The rates are based on the totals in Appendix Table H-3, which are the 
ratQS averaged across cohorts. Hence the* rates are slightly different" " 
from those that would have been calculated directly from Table IV-13, 
due to rounding error. , , 

37 



Table IV-IA 

ALUM ROCK VOUCHER DEMONSTRATION MAT READING MONTHLY ACHIEVEMENT 
RATES IN GRADE- EQUIVALENT MONTHS FOR TWO TIME PERIODS 

_ Monthly Achievement Rate s 



I II 

Sample Fall to Spring Fall to Fall 

Grade 1 (n=665) 0.6 . ^ 0.4 

Grade. 2 (n=582) 0.8 0.6 

Grade 3 (n=781) 1.0 ^ 0.7* 

Grade 4 (n=832) 1.0 0.7 

Grade 5 (n=842) ' ' 1.0 ^ 0.5 

Grade 6 (n=728) 0.8 0.7 

Grade 7 (n=8l'3) .1.2 ,0.9 



reach or exceed the month-f or-month standard during the fall-to-spring 
period. During, the fall-to-fall period, the range of rates is only 0.4 
to 0.9 'month per month; none of the samples reaches the standard. 

For students with scores for all five test administrations, Table 
IV-15 prej^ents the grade-equivalents associated with each of the five 
standard score means. Th^ pattern seen above for annual growth is also, 
reflected in this 2-year sample.. The differences betWeeti the spring and 
fall scores, for both years (Column III minus Column II and Column V minu 
Column IV) are very small; in fact, there is <no difference in 6 of the 
12 comparisons. The largest difference is an increase from spring to 
fall of 3 month3 (Grades 5-6, second sununer). Consequently, .comparisons 
of achievement over the different time periods show little differeuce.^ 

We first compare the achievement gains under three periods of time 

to the standard of a 16-month gain. When the two fall-to-spring gains 

are summed (Column II minus Column I plus Column IV minus Column JII) , 

only one of the six samples, that for Grades 6-7, reaches the s.tandard. 

Overall, the range of the sum of the two fall-to-spring gains is 9 to 16 

grade-equivalent months. When the gain is calculated from the initial 

fall to the second spring (Column IV minus Column I), again only one of 

. f 
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Table IV-15 



ALUM ROCK VOUCHER DEMONSTRATION MAT READING MEANS 
IN GRADE-EQUIVALENTS FOR TWO YEARS 









Means 






Sample 


I 

Fall 1 


II 

Spring 1 


III 
Fall 2 


IV 

Spring 2 


, V 
Fall 3 


Grades 1-2 (n=147) 


1.4 


1.8 


1.8 


2.3 


. 2.5 


Grades 2-3 (n=l47) 


1.9 


2.5 


2.5 


3.2 


3.2 


Grades 3-4 (n=193) 


2.5 


3.1 


3.2 


3.7 


3.7 


Grades 4-5 (n=194) 


2.9 


3.5 


3.6 


4.3 


4.2 


Grades 5-6 (n=191) 


3.7 


4.2 


4.2 


4.6 


4.9 


Grades 6-7 (n=136) 


4.6 


5.3 


5.3 


6.2 


6.0 



the six samples (Grades 6-7) reaches the standard. .The gains for this 
period range from 9 months to 16 months. Under the fall-to-fall period 
(Column V minus Column I), none of the samples reaches the standard. 
Here the gains range from 11 months to 14 monthss. 

Table IV-16 contains the monthly achievpmenit rates for three differ- 
ent time periods based on the means in Table IV-15. When the month-.for- 
month standard is used, three of the six samples reach this standard 
during both of the fall-to-spring periods (Columns I and II). When 
these rates are calculated from the fall of the first year to the spring 
of the second year, only one of the samples (Grades 6-7) reaches the 
standard. Under the period from the fall of the first year to the fall 
of the third year, none of the Tsamples reaches the standard. These rates 
range from 0.6 to 0.7 month per mont'h. ' ' 
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Table IV-16 

. C 

ALUM ROCK VOUCHER DEMONSTRATION MAT READING MONTHLY 
ACHIEVEMENT RATES IN GRADE-EQUIVALENT MONTHS 
FOR THREE TIME PERIODS 



Monthly Achievement Rates 



Sampl 


e 


I 

Year 1 
Fall to 
Spring 


II 
Year 2 
Fall to 
Spring 


III 
Fall 1 
to 

Spring 2 


IV 
Fall 1 

to 
Fall 3 


Grades 1-2 


(n=147) 


0.7 


0.8 


0.6 


0.6 


Grades 2-3 


(n=147) 


1.0 


1.2 


0.9 


• 0. 7 


Grades 3-4 


(n=193) 


1.0 


0.8 


0.8 


0.6 


Grades 4-5. 


(n=194) 


, 1-0 


1.2 


0.9 


0.7 


Grades 5-6 


(n=191) 


0.8 


0.7 


0.6 


0.6 


Grades 6-7 


(n=136) 


1.2 


1.5 


1.0 


0.7 



Conclu sions , 

In Section II we argued that the goal of Increasing achievement of 
participants In compensatory education programs Implies that an Increase 
In achievement should persist beyond the end of the program. If a pro- 
gram does Increase achievement, it is perhaps unrealistic to expect all 
of that Increase to be maintained year after year. However, it does 
seem reasonable to expect part of the increase to be sustained at least 
through the summer following the program. If this dges not occur, those 
concerned with compensatory education programs should have this infor- 
mation. Therefore, we believe that program evaluations, and the accom- 
panying conclusions about program effectiveness, should be pased minimally 
on a fali"to-fall time period Instead of the usual fall-to-spring time 
period. 

Unfortunately, the extent to which achievement is sustained is rarely 
studied, hence little data exist that speak to the issue. In this sec- 
tion, we have presented four sets of data that permit comparisons of 
achievement and effectiveness over both a ^all-to-spring and a fall-to- 
Tall period. We have made this comparison in several ways including the 
application of three standards of success to the results under the two 
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time periods. VJhile the standards are somewhat arbitrary (in the absence 
of accurate information on "normal" growth for educationally disadvantaged 
students), these standards serve to illustrate how conclusions about pro- 
gram effectiveness can change under the different^ time periods. We also 
extended the analysis to 2 years of a program and carried out analogous 

Table IV-17 , 

ONE-YEAR EFFECTIVENESS BY PROQRAM BY GRADE AS JUDGED • 
AGAINST THREE STANDARDS FOR TWO TIME PERIODS 



10 Percentile 
Point Standard 



8-Month 
Standard 



Month-per-Month 
Standard 



Pro gram and Grade 

Long Beach DPM 
Grade 7 (n=780) 
Grade 8 (n=468) 

Santa Barbara DPR 
Grade 7 (n=102) 
Grade 8 (n=108) 

Omaha HILINC 

Grade 3' (n=272) 
Grade 4 (n=931) 
Grade 5 (n=980) 
Grade 6 (n=316) 
Grade 7 (n=l28) 

Alum Rock Voucher 
Demonstration 



Grade 
Grade 
Grade 
Grade 
Grade 
Grade 
Grade 



(n=665) 

(n=582) 

(n=781) 

(n=llll) 

(n=842) 

(n=728) 

(n=8l3) 



Fall to 

22* 
12* 

8 
7 

NA+ 



Fall to 
Fall 



5 
4 



4 
0 

NA+ 



Fall to 
Spring 

19* 
15* 

13* 
10* 

11* 

9* 
10* 

9* 

9* 



r 



Fall to 
Fall 



7 

5 

5 

8^ 

6 

6 



Fall to 
Spring 

2.8* 
2.1* 



9* 
4* 



1 

1 

1 
1 



1.5^ 
1.3^ 
1.3^ 



Fall to 
Fall 



1.2' 
1.3^ 

1.1^ 
0.7 

0.5 
0.5 
0.8 
0.6 
0.6 



NA 



NA 



4 


4 


0.6 


0.4 


6 


7 


0.8 


0.7 


6 


6 


1.0* 


0.7 


6 


6 


1.0* 


0.7 


6 


6 


1.0* 


0.5 


5 


6 


0.8 


0.7 


7 


9* 


• 1.2* 


0.9 



'^The standard has been reached or exceeded. 
'^NA = not applicable. 
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comparisons. Table IV-17 surijrnar izes the results from the four programs 

i in terms of the three standards applied: a shift of 10 percentile points, 

an annual achievement rate of 8 grade-equivalent months, and a rate of 1 

grade-equivalent month per month • Under each standard the results for 

each grade level for each program are {Presented, first based on a fall- 
• > 

to-spring period and then on a fall-to-fall period. The asterisks indi- 
cate that the standard was reached. 

We. have demonstrated that the^f all-to-fall estimates of achievement 
ate consistently, and often substantially, lowet than the f all-to-spririg 
estimates. This reflects the findings that large mean gains over the 
school year are often followed by large losses over the following summer. 
' Hence conclusions about program effectiveness can be completely reversed 
when the summe^^ interval is included in the evaluation time period. Con- 
clusions are not always reversed, however. We have presented exampl-es 
of programs that do show a sustained impact. Regardless of the conclu- ^ 
sions reached, it is important to know if a program has a lasting impact, 
and thus we conclude that evaluations should be based on a fall-to-fall 
period instead of the traditional fall-to-spring period. 

' JO*, 

\ 
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V INDIVIDUAL-LEVEL ANALYSES 

The analyses in Section IV have rested primarily on inspection of 
means — scores averaged over individuals. Making recommendations for 
evaluation practicies on the basis of mean-level analyses assumes Implic- 
itly that the pattern of the means is reflected in at least a majority / 
of individual cases. If the summer-loss phenomenon occurred because 
a small proportion of students in each sample had enormous summer losses 
rather than because most students showed losses, we would hesitate to 
argue strongly for changes in evaluation practices. Therefore, we have 
conducted a small number of Individual-level analyses to determine 
whether the pattern of the m^ans is reflected by individuals. The 
analyses are limited by time and cost constraints. We discuss first 
the proportion of students in five samples that shoV lossfes In achieve- 
ment over the summer. We then discuss the relationships between amount 
of school-year gain and amount of summer loss. Finally we discuss the 
relationships between amount of school-year gain and amount of 12-month, 
fall-to-fall gain. ^ 

Proportion of Students With Summer Loss . - , 

To invest igate^^he extent to which the summer losses shown in the 
mean test scores accurately reflect the patterns of Individuals in the 
samples, we have plotted the school-year gains against the summer gains 
(or losses) for five samples. These samples are two cohorts of Grade 7 
students In School A In Long Beach DPM, the Grade 7 sample from Santa 
Barbara DPR, and one cohort for each of Grades 3 and 4 in the Omaha 
HILINC program. As a reminder of the mean patterns found. Table V-1 
presents the means for three test points (fall, spring, and fall) 
followed by the fall-to-spring (school-year) gains and the spring-to- 
fall (summer) losses for each of the five samples. The Long Beach and 
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.Table V-1 

EANS AND DIFFERENCES FOR STUDENTS IN THE FIVE SA^IPLES 
SHOWN IN FIGURES V-1 TO V-5 



Sample 



Long Beach DPM* ' 
Cohort 3 

Grade 7 (n=109) 
Cohort 4 

Gr^de 7 (n=82) 
Santa Barbara DPR* • 

Gra|3e 7 (n=102) 
Omah^ HILINC+ 
Co/iort 1 

^^Grade 3 (n=152) 
Cohort 2 

Grade 4 Cn:^387) 




*Standard scores, CTBS. 
"'"Grade-equivalents , Gates-MacGinitie, 



Means 



;all Spring Fall 



413 459 439 
422 . 495 463 
405 453 446 



1.97 3.05 2.60 



2.54 3.60 3.16 



Gains 

Fa]|l to Spring 
Spifing to Fall 



-20 



-32 



48 



1.08 



-.45 



1.06 \ 



-.44 



\ 



Santa Barbara samples are presented in standatd scores* (CTBS) an|p the 
Omaha scores in grade-equivalents (Gates-MacGinitie). j 

Figures V-1 to V-5 contain scatterplots of individual scores!, 
school-year gain against summer gain (loss) for each of the five samples. 
The horizontal line drawn on each chart is the zero line ij;jdicating no 
gain or loss over the summer. All students whose scores fall below that 
line experience at least some loss in achievement over the summer. The 
vertical line represents zero gain over the school year. Hence, stuOTnts 
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Our preference is to use standard scores whenever jiossible since this 
is the only metric which is defined to be equal- interval ; that is, the 
distance, between any two adjacent points on the scale is the same, 
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FIGURE V-1 RELATlONSHIf^ BETWEEN SCHOObY^AR. AND SUMMER GAINS IN CTBSSTANDARD SCORE UNITS 
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FIGURE V4. RELATIONSHIP BETWEEN SCHOOL YEAR AND SUMMER GAINS iH GATES-M»eGINITIE GRADE 
EQUIVALENTS FOR OMAHA HILINC STUDENTS: GRADE 3, COHORT 1 
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FIGURE V S RELATIONSHIP BETWEEN SCHOOL-YEAR ANIX8UMMER GAINS IN GATESM«GINITIE GRADE 
EQUIVALENTS FOR OMAHA HILINC STUDENTS: GRADE 4, COHORT 2 



whose scores fall in the lower right-hand quadrant are those with gains 
over the school ''year and losses over the summer. Table V-2 following 
the scatterplots summarizes these numbers for'each sample. 



Table V-2 ' 

APPROXIMATE ^^UMBER OF -STUDENTS WHOSE PATTERN FOLLOWS THE MEAN 



Summer Loss and School 



Sample 



Long Beach DPM, School A 

Cohort 3 (n=109) 

Cohdlt 4 (n=82) 

. Santa Barbara DPR (n=102) 

Omaha HILINC 

Cohort 1 

Grade 3 (n=152) 

Cohort 2 

Grade 4 (n=387) 



Summer 


Loss 




Year Gain 


Number 


Percent 


Number 


Percept 


85 


78% 


80 


f 

73% . 


73 


89 


71 


87 


56 


55 


52 


51 


108- 


71 


105 


69 


258 


67 


251 


65 



TJie' findings are encouraging in terms of generalizing fincftngs at 
the mean level to individuals. In' all five samples, at least 50% oi 
the students follow thfe pattern of the mean. In four of the five 
samples the proportion of, students with school-year gains and summer 
losses is at least 65%. The sample with the lowest percentage following 
the pattern, Santa Barbara with only 51%, is also the sample with the 
smallest summer loss' at the meaa level (see Table . Therefore, we 

conclude that the phenomenon of sun\mer losses at the mean level is not 
the rej?ult of a small number of extreme cases but rather reflects the 
pattern of the majority of students in. each sample. 



School— Year Gain Versus Summer Gain 

— - — ^ - — " "~ ' < 

We were next interes^d in whether any relationship existed between 
the amount of gain achieved during the school year and the amount lost 
over the summer. In other words, were students with large school-year 
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gains more or less likely than those with small school-year gains to 

4 

have large losses over the sumni^r. The scatterplots already presented 
±vt Figures V-1 through V-5 suggest that there is such a relationship; 
students who gain a lot over the school year tend to lose a lot over the 
summer and, conversely j those whp g^in little over the school-year lose 
little over the summer. These relationships are summarized by the 
correlation coefficients in Table V-3. • . 



. [Table .V-3- . J. ' ■ , 
CORRELATION BETWEEN ScioOL-YEAR GAINS AND SUMMER GAINS 



Sample 



Long Beach DPM 
. Cohort 3 (n=109) 
Cohort A' (n=82) 
Santa Barbara DB.Jl (n=102) 
Omaha HILINC 

Cohort 1 (n=152) 
Cohort 2 (n=387) 



Correlation 
Coefficient 



-.3A 
-.30 
-.A9 



-.72 
-.60 



All the correlations are negative (and statistically significant 
at the .01 level), indicating that large school-year gains tend" to be 
associated with large summer Id'ssei^ and, the converse. Since these 
correlations are between two nonindependent gain scores (Spring minus 
Fall 1 and Fall 2 minus Spring), they are necessarily fraught with ' 
error. -However, the size of the correlations- suggests that there 
might be a real relationship, albeit inflated by measurement error. 
To determine if this were the^case, we calculated a rough estimate of 
the expected correlation between two gains based on error alone. That 
is, we assumed that there was no correlation between school-year and 
summer gains and calculated the correlation using an estimate df the 
reliability of the tests. These calculations resulted in a correla- 
tion of approximately -.2 on the assumption of no true relationship. 
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Consequently, we concluded that our correlations, all of which exceed 
(in an absolute sense) -.2, ,,do reflect a true relationship , but probably 
one that rallies ; fy^O^ ^ 1- tio;;^ - 3 fti^is'£ea4 .o f -.3 to -.7. Nevertheless, 
in the world of educational r*esearcny»,:fcot'r^lat ions as high ^ -.5 are 
rare. ' r * 



School-Year Gain Versus 12-Month Gain 

■ • ■ — ■ i.- 



If Students with large school^-ye^tr gains' are likely to have large 
summer losses while those with small^'^-fechooi-^year gains will have small 
summer losses^ then the next question of interest is whether; the 
differences in amount of summer loss .are sub^tointial enough^ to alter 
a student's relative position by the end o^]^,?tne slimmer. TKatlis, Is 
the perpentagC^ of school^^ear gain that is' losO J\igh«r..*^Qr, studejits 
with large school-year gains ttian for.: thds^ with l^V e^tloidl?^ 



If this ^ere true, judging students on the ,b^§|^^.^^. a spring 'scare • 
, wduld be tremendously misleading — notA 6)tily ■ becau3§^t3re^ ^'flall'^'^ wOiild 
be* lower but also because the relative Tanking of ^fe^ud?fnts, woU^ Qh^p^Z^^* 
If this is not the cas^, however, the ranking of stu3^i)ts ^Voul^^^.r^ 
the same-^those' wi th the highest school-year, gains would a^^O/.^r^Ve th^* * 
• ^^ighest 12-month, fall-to-fall gains. .The high school-year g'alner^ 
might lose more than low school-year gainers, but have more to lose; 
thus, they might remain at the top of the distribution of 12'-i][ionth , 
fall~to-fall gains. • . . ' ! ' 

r * 

% - ' ^ 

To test this hypothesis, we performie'd twp similar afial3^ses. First, 
we divided the school-year gains into seVen intervals for the same five 
If/ samples analyzed in the preceding dlscijssion. We then calculated the 

mean of > the 12~pionth (fall-to-fall) gains for all students falling in 
each interval. The results for Cohorts 3 and 4 in Long Beach, School A 
and Yor Santa Barbara are presented in standard scores in Table V-4 , 
along with the grade-equivalent results for Cohort -1, Grade 3 and* Cohort 
2, Grade 4 in Omaha. 



For all five samples there is a clear ^trend for the fall-to-fall 
gains to increase as the school-year gains increase. With a few minor 
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Table V-4 

MEAN GAIN OVER 12 MONTHS (FALL TO FALL) BY SIZE OF SCHOOL-YEAR GAIN 



Omaha HILINC 

Cohort 1, Grade 3 

Cohort 2, Grade 4 



< 0 



-0.01 
(n=18)" 

-0.01 

(n=32) 



Means GMn. In Grade-Equivalents 



0.1- 
0.5 



0.44 
(n-30) 

0.33 
(n»75) 



0.6- 
1.0 



0.59 

(n»36)- 

0.52 
(n-98) 



1.1- 
1.5 



0.58 
(n-32) 

0.7f 
(n»83) 



1.6- 

2.0 ' 



0.92 
(n«^15) 



2.1- 
2.5 



1.04 
(n-8) 



0.94 0.94 
(n-56) (n-2'8) 



Sample 


Mean 


12-Month 


Gain in 


Standard : 


Scores by 


School Year Gain 


< 0 


1-20 


21-4J) 


41-60 


61-80 


81-100 


>101 


Long Beach DPM, School A 














✓ 


Cohort 3 


-13.3 


1.0 


11.6 


19. « 


49.2 


51.7 


94 




(n=13) 


(n=14) 


(n=22) 


(n=18) 


(n=25) 


(n=ll) 


(n-6) 


Cohort 4 


-22.0 


-25.2 


6.9 


29.9 


27.6 


55.1 


91.7 




(n=3) 


(n=4) 


(n-r2) 


(n=15) 


.(n-13) 


(n»17) 


(n-18) 


Santa Barbara DPR 


14.8 


14.4 


28.9 


38.9 


58.5 


96.3 


81.7 




(n=12) 


(n=14) 


(n=19) 


(n»23) 


(n=17) 


(n-3) 


(n»14)' 



>2.6 



1.54 
(n-13) 

1.52 
(n-17) 



0 



62 



exceptions, these figures suggest a strong relationship between amount 
of school-year gain and amount of 12-month gain* 

We then produced scatterplots of the relationship between school- 
year gain and 12-month gain to verify the findings from the first 
analysis. The scatterplots confirmed the relationship. They are 
summarized by the correlation coefficients shown in Table V-Sj'^all of 
which are significant at the .001 level. 

We conclude that even though students with large gains over the 
school year have large losses over the summer, the losses are not 
proportionately larger than those for students who have small gains 
over the school-year. Therefore, the- ranking of students by size of 
gain at the end of the school year is similar to their ranking at the 
end of the following summer. 

In 'conclusion, ' we find that the pattern of the means found in the 
analyses in Section IV is reflected at the individual level. This 
makes us feel more secure in making recommendations for evaluations^ 
conducted at "a group level. Additionally, we suspect that there is 
an interesting relationship between amount of school year gain and 
amount of summer loss, as well as between amount of school-year gain 
and 3 2-month (fall-to-fall) gain. While our analyses are only a 
beginning in this line of investigation,* we think the initial findings 
are of sufficient Interest to suggest pursuing this line of research. 



We also examined correlations between initial fall score and school- 
year gain but found no significant relationships. 
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Table V-5 



CORRELATION BETWEEN SCHOOL-YEAR GAINS AND 
12-MONTH (FALL TO FALL) GAINS 



Long Beach School A 
Cohort .3 (n=109) 
Cohort 4 (n=82) 

Santa Barbara (n=102) 

Omaha HILINC 

Cohort 1 (n=152) 
Cohort 2 (n=387) 
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Correlation 
Coefficient 

0.69 
4,^. 0.81 
0.52 

0.58 
0.48 



VI SUMMARY AND RECOMMENDATIONS 

Summary . - . 

~ Increasing the achievement: of educationally disadvantaged students 
is a widely shared goal of compensatory education. This goal implies 
that increafses in achievement can in some way affect the futures of dis- 
advantaged children by equipping them with skills equivalent to those of 
their more advantaged peers. If increases in achievement are ephemeral, 
this goal will not be realized. Therefore, we have argued that judgments 
of the effectiveness of compensatory education programs should include 
measurement of the extent to which the program impact is lasting. 

Only a very few studies of compensatory, education have investigated . 
' the issue of sustained effects, and most of these are restricted to pre- 
school programs. Since we could not draw on previous research, we turned 
our efforts to reanalyzing previously collected evaluation data; data 
that would allow estimates to be made of a sustained program impact. We 
obtained and analyzed data from four differetvt compensatory education 
programs. 

♦ 

The primary finding of these analyses \3 that conclusionsf^ about 
program effectiveness, regardless of what standard is used, are greatly 
influenced by the period of time over which the program ±s judged. Spe-- 
cifically, we show that the inclusion of the summer months in the evalua-. 
tion can substantially reduce estimates of achievement and often reverse 
positive judgments of program effectiveness. This results from* the fact 
that losses in achievement often occur over the summer. In three of the 
four data sets presented, gains during the school year were followed by 
losses over the s.umroer. In the; fourth, although there was not an actual 
achievement loss over the summer, there was a r^ductioii" in rate of ' ' 
achievement. 



Additiopally , ,we demonstrate that different standards for success 
can result in different conclusions about program effectiveness. We 
have not explicitly compared the standards to each otlt^er since our pri- 
mary interest was the effect of tbfe time period for each standard. 
Nevertheless, we showed that the 10-percentile-point standard is mpre 
•stringent than the two standards which entail grade-equivalent scores 
and thus is less likely to be met, especially during a 12-month, fall-to- 
fall evaluation. 

Finally, the ,'lBXtent to which individuals in each sample follow the 
pattern discovered in the means was investigated. In the five samples 
studied, the achievement patterns of a majority of the individual stu- ^ ^ 
dents were the same as the*^attern of the means. We conclude, therefore,' 
that the consistent finding of school-year gains and summer losses is 
not a function of a small number of individuals in the sample with large 
summer losses, H 

As a last step, we lopked at the relationship between the size of ^ 
the school-year gain and the size of ^the summer gain (usually loss) f-^^^ 
individuals. Although the correlations describing this relationship 
fraught with measurement error,^ they were sufficiently large to convin 
US tha^t there is an association between amount of school-year gain and 
summer loss. Specifically, students who gain the most over the school- 
year tend to be those who lose the most over the summer,' However, 
an^il^ses of the relationship between school-year gain and 12-month gain 
suggest that the ranking of ^students by size of gain does not shift 
dramatically from the end of one school year to the beginning of the 
next . 

These data represent the only attempt to address the i^sue of sum- 
mer loss with several longitudinal data sets, thereby eliminating the 
c'onfounding introduced by cross-sectional data. Although we analyzed 
only four data se'ts, th^y represent different programs, different age 
leVel^ different subjects, .different tests and many different schools. 
Since the findings of summer losses are quite consistent across all of 
these variables, we suspect that our conclusions are not limited to these 
four programs. Combined with que^ions raised by pr^vidVis research, such 

^ 
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as the inconsistencies between school-year evaluation results and the 
results of annual state-wide testing programs, we suspect tha^ the 
existence of summer losses, is quite common for educationally disadvan- 
taged students. Therefore, we urg6 that this phenomenon be taken into 
account in designing and carrying out evaluations of compensatory 
education programs. 

It should be noted that our data demonstrate that programs can 
show evidence of sustained effects. Hence, a longer evaluation time 
period does not imply that all programs would-be judged ineffective. 

Recommendations 

ESEA Title I programs are usually evaluated on the basis of fall 
and spring test scores for a given year or a spring only score (some*- 
times using the previous spring score as a pretest). For districts 
that administer tests both fall and spring, ^ur recomi^endati(Jn is not 
to change data collection practices , but rather to include analyses of 
students over the 12-month, fall-to-fall period. For dis t ric ts %hat 
administer tests in t\\e spring only, we sugge 
lection. If only one test can be administere 
that this be done in the fall, thus permittin 
fall achievement. When a pragram is evaluate 
tp-spring' scopes , the results are perhaps not 
based on a fall-to-spring period since one summer is included in the 
spring-to-spring period of time, however, from a logical perspective, 
one should- look for sustained gains ^ some time after participation in 
the program. Therefore, evaluating a program from one^ spring to the 
next does not reflect the extent to which gains have been sustained 
after the program.. 

We are particularly concerned over the practice of "graduating" 
students from ^l^rogram on the basis of a spring test score. When a 
district uses a spring score f or-'determining program eligibility,* 



*This practice exists in -many districts,, but we have no information on 
how widespread the practice is on a national scale. 
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sL-dv cnange in data coi- 
£ annu^^J43r, - we recommend 
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g analyses 
d on the basis of spVing- 
as misleading as thos^ 



students who attain their "expected** grade-level score .are no longer 
eligible for program participation* Since some of the achievement g«i^ 
reflected in- the spring score may be lost by t^Jie end of the "summer, 
extreme care should be taken i'^ assuming that a spring scoi:e accurately 
reflects a student-^'s achievement level . We urge, therefore, that fall- 
scores rather than spring scores be used as a t5asis for judging eligi- 
bility for the program. ^ . p> 

While we are willing- to make suggest ions ^poncerning appropriate 
evaluation strategies, we are not willing to draw conclusions about the 
causes and therefore possible solutions for the summer loss phenomenon. 
Our recommendations are concerned with providing valuable information 
t6 program personnel about sustained achievement gains. We hoip^ that 
this would be a first step In understanding why summer losses occuf* 
If, for example, the phenomenon Is a function of the measures used 
(the standardized achievement tests) one would want to change the 
measures instead of the program. If It Is a result of instructional , 
techniques that militate against retention, then the techniques should 
be changed. Without additional information on the causes, it is 
dangerous to suggest alternatives such sts a different school calendar 
or summer school program. Therefore, the next step In this line of 
research should concent^rate on -eicplaining summer losses and relat^n- 
ships at the individual level* Only at this point can one , recommend 
an appropriate remedy without the risk of exacerbating the situation. 
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Appendix B ^ 
REVIEW OF RESEARCH ON "EXEMPLARY," PROGRAMS 

•J 

To meet the 'specifications of the 196^7 'Amendments td'the Elementary 
ai^d Secondary Education Act (ESEA) , American Institutes for. Research 
(AIR)^.was requested Ito identify, selectr, analyze, and describe Title I 
and ^compensatory etlucat^^on programs for disadvantaged children, from 
preschool thrpugh grade 12, which yielded benefits in terms of cognitive 
achievement. It was expect^^i that the in^^ormation gleaned from the AIR 
reports would enable LEA*s to improve their own .programs , or p6ssibly' 
even duplicate a program that had already been judged a success. The 
programs. AIR^identif iad were to be used as examples for the entire 
nation. - the thlr^e documents that resulted from this inquiry constituted 
the- basis for this phase of our inquiry: ' 

^ '^^^ ' ^ 

Hawkridge, D.^ G.^ A. G, Chalupsl^., and A.^.H. Roberts,^ "A Study ' 
' of Selected , Exemplary" Programs fot tSe Education of Disadvantaged 
Children," Paid Alto, Call/o|*nia, America^ Institutes foi; Researchr'*''' 

^ (September -1968).. (ED'023777) < ' ^ 

* * ' , ' " . . ^ . ■ .-^ 

Hawkridge, G.^iV. L.' Campeau , K. M. DeWltt^ and P, K. 4 '. 

Trickett, "A Study .of^v 'Further Selected Exemplary Programs for < 
* the Education -df Dls3<iVantaged Children," .Palo Aito , Calif ornia , 

American .Ii]stitutes for Research (June 19.69). ,^ED 036668) 

Wargo, M. J., P. L*. Campeau, G. K. Ta^^lmadge, "Further Examtna^ ; " 
tion pf Exemplary Prograij^ for E^ucaMng Disadvantaged Children 
Palo Alto, (?klifornia, American Jrtstrfcn^s for Research (July 
/971). (ED 055153) 



The 1968 ^and, 1969 reports , dbciiment" 31 exemplapy programs. The X971 
report describes 10 ne^ly' identified exemplary prograniis.' Additionally^ 
the th^en ct^rerf^ status ^of the .^Viginally IdenttLf^led progr^Tijs Were 
teexamined. -We cpntacted D^i A* G. Chalupsky by phone to determine 
whfether^more receht and updated inqu^iry ha4 been riia^e by AIR. To the 
best of his Jpowled^e, he Va§ a\^are 'of. no suctt recent' inquiry. > ^ 
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We reviewed the 41 programs in this final report in order to deter- 
♦ 

mine -which programs appeared most likely to have data meeting our cri- 
teria. Twelve of the 41 programs were ^preschool programs and could be 
eliminated immediately as being outside the scope of our study. Fourteen 
more programs were eliminated for one or more of the following reasons: 
the program was bilingual and, therefore, outside our scope of work; 
"e^alua^ion of the program had ceased by 1971; or the program had been 
incorporated into a larger effort such as Follow Through. This left 15 • 
programs which -we then investigated. 

As far as could be determined, from descriptions and evaluation 
reports, none of the 15 remaining programs met our criteria for measuring 
sustained effectiveness. However, in' some cases it could be inferred 
that such information could be made available. In other cases, where 
such inference vah not obvious, the project's ccjntinued success warranted 
our inquiry as to whether data on'sustained ef &ectivene5s existed or 
could be derived. To obtain additional information, an attempt was made 
tOjf contact each proj(L'Ct. It was immediately established that 8 of the 15 
programs were no Longer in existence and therefore^ could not be con- 

\ 

sidered as candidates. The remaining 7 programs are listed below with 

the reason for eliipinating each: 

* - 

• Remedial Read:^ng Laboratories, a Title I funded program did 
'administer st^l^dardized achievement tests in the fall; and 

^ spring of each year. Howevej , the assistant superintendent 

of the district expressed an unwillingness to relea^e^ the 
data for reanalysis and our repeated requests for information 
went unanswered.. . " ^ 

is' 

• Project Mars^ aTso a Title I program with annual fall and 
spring standardized achievement tests, sent us clasa sheets 
with crossr-sec tional data /or each year of their program. 
However , -when St ud^nts were matched to create longitudinal 
dataV the sampj^e of students with more than 1 year^^s data was 
too small to support t^ean^Jl^^sis . 

• Colleg'e Boajfd is also a Title I program. While considerable 
• data" exist from annual fall and spring t est ^ administrat fons , 

lo^gitudinal^^f lies are not maintained on individuals. We 
determined that /deriving longit\idinal data for even a subset • 
Mf students would be "^prohibit iV^ly expensive^ 

• Project R-3 is a California Demonstration project in Intensive 
Instruction in 'Reafiiftg and Mathematics.' The data from this \ 

V ■ ■ ^ ■ . ' " 



project seemed at first to meet most of bur criteria. 
Pretests and posttests were annually aclministered in Grades 
, 7, 8, and ^. Unfortunately, when the data arrived, so many 
data were missing that reanalysis cou^d not be attempted, 

• Project Conquest is another Title I project.. The combination 
of its having standardized tests ^the spring only and very 
small sample sizes in each grade e^^ed us to eliminate the 
project. ' 

* w 

• Higher Horizons 100 is only a 1-year program for which no 
^ follow-up i^dat a were available. 

N 

• Intensive Reading Instructional T^ams is a program spanning 
less than 1 full school year. ^ " * , 

In summary, of the 15 programs identified as candidates for re- 
analysis, 8^ were immediately eliminated since they no longer existed, ' 
6 did not have data that wfculd support rean^lysis and 1 progr^, which 
might have met our criteria, was prohibit ivelV expensive. 

We then extended our Inquiry by invest igafsjng a recent and extensive 

effort to identify successful programs for ProjectNLnformat ion Packages 

(PIPs). Of the 6 projects identified for PIPs, 4 had previously been 
# 

reviewed by AIR (the last 4 of the projects describe\i above). This 
left only two candidates for further investigation. The first. Project, 
Catch-up, is a Title I project that administers standardized achievement 
tests only in the spi:ing. This^**comSined with a small sample size forced 
us to eliminate the project-. The se>cond. The High Intensity Tutoring 
Center, is also a Title I prfeject involving students in grades 6-8 as 
tutWs and tutored. The roles of the participants change throughout the 
program and the ^length of time in the program varies. , Therefore, in ' ' 
light gf the expense involved in obtaining yat'^a, we eliminated this 
project. - . . 

. ' Overall, then,. what promised to.be the most likely source of, 
adequate data produced nothing. Our other approaches were slightly more 
fruitful. . . ' ^ 
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Appendix C 
ACQUISITION AND PREPARATION OF DATA 

After our lengthy search for data and affcer obtaining permission ^o 
reanalyze the data described above, we were faced with the t^sk of trans- 
forming the data into a form amenable to reanalysis. While the data had 
been carefully selected according to our criteria, we did not liqiit our- 
selves tQ data already in computer-readable form or to data with longitu- 
dinal records for individuals. Had these limitations been imposed, all 
dat'a would have been eliminated. The purpose of this section is to 
briefly acquaint the reader with the steps involved in the process of 
preparing locally collected evaluation data for reanalysis. 

For two of the programs, the data were available only on handwritten 
lists that contained student identifiers and scores, sepai'at'ely , for each 
test administration and grade level. Preparation of these data began 
with matching students td create longitudinal ' data . The data were then 

ter^d on coding sheets, keypunched, and recorded on computer tape or 
disc. ; Corrections were constantly incorporated throughout t;Jhis. process, 
as changes, such as errors^ in recorded scores or identifiers and addi- 
tional scores, were received from the distrit?ts. While we now have 
confidence in the accuracy of the analysis tape, the process of reaching 
this stage was time consuming and tedious. 

Three districts provided data on computer-readable tape. For all 
' of these, the data were arranged cross-sect ionally and had to be com- 
pletely reformatted _ for longitudinal analysis. While this was not ap . 
overwhelmingly difficult task for two of the tapes, it did require ' ^ 
repeated clarification 'bt the coding and organization of t>ie tape since 
^written documentation is rarely adequate. The thir^jt*^ posed much 
more serious difficulties.. As a test for accuracy, ve duplicat^ed J^or each 
data set the cross-sectional analyses that the districts had carried 
.out. In the caae of this tape. It was evident that it conta^ined a 




number of errors. This was verified through conversations with the 
district. We requested a corrected tape, which, after many months 6f 
reminders, finally arrived — almost 1 year after the initial tape was 
received . Unf ortonately , the "corredred** tape contained even more 
errors than its predecessor and thus could not be used. ^ We therefore 
could not, Vith any degree of confidence, reanalyze the data and were 
forced to eliminate the program from our study. 

Data from the remaining program were stored on a combination of 
computer tapes and floppy discs. Since floppy disc^ cannot^ be read by 
standard computer systems, we requested printouts for all data stored 
in this way.. The data from the printouts then had to be key-punched 
and merged with the data from t^ie tapes. 

We have documented these problem^ not as a criticism of the dis- 
tricts involyed but as a guide to those 'who may find themselves involved 
in similar work. In general, we received tremendous cooperation from 
the districts. We feel that this cooperation was in part due to our 
having visited the program^ and our willingriess to share oilr findings 
with the districts. 
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Appendix D ^ 

ADDITIONAL DATA FROM THE LONG BEACH 
DEMONSTRATION PROGRAM IN MATHEMATICS 

Appendix D containi^Hldditional data from the Long Beach Demonstra- 
tion Program In Mathematics (DPM) . Table D-1 describes the cohorts for 
which we have data by year and grade level. Each cohort represents a 
group of students who progress through school together. 

Table D-2 presents the means and gains in three metrics (standard 
scores, grade-equivalents and percentile scores) for samples of students 
with at least three consecutive yii^gt points by grade, school, and cohort 
The totals for grade equivalents and percentiles, which are averaged 
across school and cohort, are the means presented in Table IV-1. Table 
D-3 presents the monthly achievement, rates in grade-equivalents over 
two time periods for the same samples of students. The rates for each 
school and cohort^re based on the means presented in Table The 
totals for each grade^are those preaented in Table IV-2. 

Table IH4 presents the means- in three metrics for the sample of 
students with at least five consecutive test p7)ints by^ school and ^^ohort. 
The totals for g^rade equivalents and percentiles also appear in Table 
IV-3. For the s^me 'samples of students. Table D^ presents monthly 
achievement rates in grade-equivalents for three time periods. These 
totals are also in Table IV-4. 

For comparison, we also include a table of cross-sectional data. 
The means in three metrics for all students tested at a given point are 
presented by school and cohort in Table D-6. 
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Table D-1 



LONG B£ACH DPM COHORT DESCRIPTION BY YEAR AND GRADE 



Sample 




' 1969- 
1970 

Cohort 1 



1970- 
1971 

cohort 2 

Cohort 1 



1971- 
1972 



1972- 
1973 



Cohort 3 Cohort 4 
Cohort 2* Cohort 3 
Cohort 1 Cohort 2 



1973- 
1974 



Cohort 4 
Cohort 3 



''Data vere not available for this year .for Cohort 2. 
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Table D-2 

MEANS AND GAINS OVER TWO TIME PERIODS IN THREE METRICS FOR STUDENTS 
WITH AT LEAST THREE CONSECUTIVE TEST POINTS BY GRADE, SCHOOL, 
AND COHORT— LONG BEACH DPM, CTBS MATHEMATICS 







Means 




Gains 




I 


II 


III 


IV 


V 


Sample 








Fall to 


Fall to 


Fall 


Spring 


Fall 


Spring- 


Fall 


Grade 7 










School A 












Cohort 1 ('n=170'i 












Sf';^nH^T*H o r» t* o o a 




/ C "7 


452 


34 


* 

29 

- 


oLciiiuara cevlaClon 


(6U) 


(58) 


(77) 










6.0 




0.8 


Percentile 


1 Q 


9 

Z J 


zO 


/• 


4 


Cohort 3 (n=109) 












Standard •^corpQ 


U 1 




/. 9 n 


46 


26 


'-'Udii^dLU UcVXclLJ.OII 


( 




(83) 






oLduc t:t|uivaienL 


D . U 


6 . z " 


5,6 


1 . Z 


0.6 


pp rr pn ^ 1 1 p 


1 ^ ' 


ZD 


io 


11 


1 


Cohort A (rt=R?) 














A 9 9 


/. Q Q 


453 


71 




Standard deviation 


(60) 


(82) 


f81 ) 




Grade equivalent 


5.2 


7.3 


6.3 


2.1 


1.1 


Percentile 


18 


43 . 


24 


25 


6 


School B 












, Cohort 3 (n=241) 












Standard scores 
^ Standard deviation 


445 


523 


489 


78 


44 


(60)- 


(72) 


(72) 






Grade equivalent 


5.'S 


8.1 


7.1 


2.3 


1.3 


Percent ile 


28 


57 


35 


29 


7 


CoTiort 4 (n=178) 












Standard scores 


^ 454 


536 


502 


82 


^-48 


Standard deviation 


(57) 


(68) 


(77) 






Grade equivalent 


6.1 


8.4 


7.5 


" 2.3 


1.4 


Percentile 


32 


63 


40 


:>i 


8 


Total Grade 7 (n=780) 












Standard scores 


435 


500 


474 


65 


39 


Standard deviation 


(62) 


(78) 


(80) 






Grade equivalent 


5.5 


7.4 


6.4 


X.9 


1.1 


Percentile 


23 


45 


28 


22 


5 



Based on -January test administration. 
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Table. D-2 (Concluded) 



Sample 



Grade 8 « 
School A 

Cohort 1 (n=124) 
Standard scores 
^ Standard deviation 
Grade equivalent 
Percentile 
Cohort 3 (n=68) 
Stancjard scores 
Standard deviation 
Grade equivalent 
Percentile 
* Cohort 4 (n=89) 

Standard sc6r^s 
Standard deviation 
Grade equivalent 
Percent^ile 

School B 

Cohort 3 (n=l87) 
Starfdard scores 
Standard deviation 
Grade equivalent 
Percentile 

Total Grade 8 (n=468) 
Standard scores 
Standard deviation 
GradQ equivalent 
Percentile 



I 

Fall 



Means 
II 

Spring 



Gains 



III 
Fall- 



IV 
Fall to 



V 

Fall to 
Fall 



A AO 


514 


502 


54 




(78) 


(78) 


(71) u 






6 • 2 


/ • o 




1.6 


1.3 


23 


J 1 




14 


3 


/. A A 


*-t 17 7 


496 


53 


50 


(80) 


(95) 


(89) 






5.8 


7.4 


7.3 


1.6 


1.5 


18 


31 


24 


13 


6 


451 


492 ' 


485 


4i 


34 


(81). 


(87) 


(80) 




0.9 


6.0 


. 7.2 


6.9 


1.2 


20 


28 


20' 


8 


0 


489 


541 


542 


52 


53 


(81) 


(83) 


(75) 






7.1 


8.6 


8.6 


X.5 ■ 


1.5 


35 


48 


42 


13 


7 


468 


518 


514 


• 

50 


46 


(82) 


. (86) 


(80) 






6.4 


7.9 


7.& 


1.5 


1.4 


26 


38 


30 


, 12 


4 

• 



Notes 

1. 
2. 
3. 



Standard " scotves ara- publisher ' s standard scorer.. , 
Standard deviation is standard deviation of standard scores. 
Grade equivalent is that associated with standard score mean 
in years. 

Percentile is the percentile associated with the standard 
score means. 
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Table D-3 



MONTHLY ACHIEVEMENT RATES IN GRADE EQUIVALENTS OVER 
TV70 TIME PERIODS FOR STUDENTS WITH AT LEAST THREE 
CONSECUTIVE TEST POINTS— LONG BEACH DPM, 
CTBS MATHEMATICS" 



Mon thly Achievement Rate 



Sample 



Grade 7 * 
School A 
Cohort 1 
Cohort 3 
. Cohort 4 

School B 
Cohort 3 
Cohort 4 



(n=170) 
(n=109) 
(n=82) 

(n=241) 
(n=178> 



Fall to Spring 



2.3* 

1.7 

3.0 

3.3 

3.3 



Total Grade 7 (n=7B0) 



2.8 



t. 



11 

Fall to Fall 



1.3* 

0.6 

1.1 



1.3 
1.4 

1.2' 



Grade 8 
School A 
Cohort 1 
Cohort 3 
Cohort 4 



(n=124) 

(n=68) 

(n=89) 



School B 

Cohort 3 (n=187) 

Total Grade 8 (n=468,^ 



2.3 
2.3 
1.7 

2.1 
2.1 



1..3 
1.5 
0.9 

1.5. 
1.3 



.Rates calculated from, January of seventh grade. 

These rates are averages of the sample rates. They are not based on 
the means in Table IV-2 since the interval from fall to spring differs 
for School A, Cohort 1. , ' ^ 



Table D-A 

MEANS IN THREE METRICS FOR STUDENTS WITH AT LEAST FIVE 
CONSECUTIVE TEST POINTS BY SCHOOL AND COHORT 
LONG BEACH DPM, CTBS MATHEMATICS 



Means 





I 


1 1 


III 


IV^ 


V 


c>anipie 


7 "C 1 1 

/ r al 1 


/ Spring 


0 rail { 


3 Spring 


9 Fall 


School A ' 












Cohort 1 (n=106) 












Standard scores 


421 


462 


458 


- 508 ■ 


494- 


Standard deviation 


(59) 


(66) 


(73) 




(68) 


Grade equivalent 


5.2 


6.3 


6.2 


7.7 


7.2 


Percentil>e 


^ 18 


28 


22 


34 


23 


Cohort 3 '\n=^5D 






- 






Standard Scores 


428 


478 


454 


509 


499 


Standard deviation 


(69) 


(83) 


~<82) 


(94) 


(96) 


Grade equivalent 


5.4 


6.7 


6.1 


7.7- 


7.4 


Percentile 


20 


35 


21 


35 


25. 


Cohort 4 (n=67) 










V 


Standard scores 


419 


491 


461 


500 


493 


Standard deviation 


(62) 


(83) 


(84) 


(88) . 


(81) 


Grade equivalent 


5; 1 


7.1 


6.2 


7.4 


7.2 


Percent ile 


17 


4l- 


23 


. 31 


23 


School B 












Cohort 3 (n=153) 
Standard p CO res 












451 


526 


496 ■ 


• 545 


* 54^ 


Standard deviation 


,(61). ' 


(76) . 


(79) 


(84) 




Grade equivalent 


6.0 


8". 2 


7.3 


8.7 


8\7 


Percentile 


30 , 


58 


38 


50 • 


44 


Total (n=378) 












Standard scores 


• 434 


495 


473 


522 


516 ■ 


Standard deviation 


(63). 


(80) 


(81) ■ 


(86) 


'(82) 


Grade equivalent 


5.5 


7.3 


6.6 


7.9 


7. 7 


Percent ile 


23 


43 ♦ 


28 


40 


- 31 



Based on January testing. 
Notes 



1. Standard scores are publisher's standard scores. 

2. Standard deviation is standard deviation of standard scores'. 

3. Grade equivalent is that associated with standard score m^an 
V ill years. ^ ^ 

4. Percentile '-is the percentile associated with the standard^ 
score means . . i * . 
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Table D-5 

MONTHLY ACHIEVEMENT fATES IN GRADE EQUIVALENTS FOR THREE TIME PERIOd's FOR 
STUDENTS WITH AT LEAST FIVE CONSECUTIVE TEST POINTS— LONG BEACH DPM, .CTBS MATHEMATICS 



Sample 



School A 
Cohort 1 
Cohort 3 
Cohort 4 



(n=106) 
(n=52) 
(n-'67) 

V 

(n=153) 
Total (n=378) 



School B . 
Cohort 3 



I ' 
Grade 7 
Faj.1^ to Spring 



2.8 
1.9 

2.9 

3.1 
2.8 



Monthly Achieveiaiwit Rates 



II 

Grade 8 
Fall to Sprinj j ; 



2.1 
2.3 
1. 7 



?.0 

2.0 



/ III 
G/ade 7 Fafl to 
rade 8 Spring 



1.9 
1.4 

1-^ 




1.6 

r. 6' 



IV . ' 
Grade 7" Fall to 
G^ade 9 Spring 



1.3 
1.0 
1.1 




The rat^s in Columns I and II- were calculated by 'dividing the. fall-to- 
spring gain by seven (the number of months between tests). The rates 
in Column III are- the fall Tth grade to spring 8th grade gains divided 
by 17 and the Column IV rates are the fall 7th grade "to' fall 9th grade 

gains divided by 20 {lb grade-equivalent months'for each year). 

• . ' ■ ■ ' ' 

Fates, calculated f roifr-^aftuary of _Gra<}e 7. . ' ^' '' \ 



I 

Table U-6- . 



'CROSS-SECTlONAi- MEANS AND STANDARD DEVIATIONS IN STANDARD 
SCORES FOR ALL STUDENTS TESTED—LONG BEAGH DPM, 
CTBS MATHEMATICS J 



_ Sample 



School A 
Cohort 1 

CroBs-sectlonal meand 
Standard deviatlona 

Number 
Cohort 3 

Cross-sect lonal means 

Standard deviationB 

Number 
Cohort 4 ' 

Cross-sectional means 

Standard deviations 

Number 
Schpol B 
Cohort 3 

Cross-^sect ional means 

Standard deviations 

Number 
Cohort 4 * 

Cross-sectional means 

Standard deviations 

Number 

Total 

Cross-sectional means 
Standard deviations 
Number 



'.ERIC 



Grade, 7* 


Grade 8 


^Grade 9 


.Fall 


Spr Ing 


Fall 


Spring 


Fall 


t 

A21 

(59) .' 


A55 
(68) 


A52 
. ,(77)- 


>. 511 

<79), 
Vn^lov) 


A91 
' (73) ^ 
(n*166) 


A13 
(61) 
(n»165) 


A58 
(77) . 
(n-173) 


A35' 
(80) 
(i\-lA7) 


A79 
(91) 
(n-X27) 


A97 
(8A) 
(n-98) 


A18 
(58) 
(n-lAA) 


A83 
(83) 
(n=»lA9) 


A53' 
(81) 
(n°115) 


A9A 
(86) 
(^-116) ' 


A69 • * 
(82) ■* • 
(n»121) 


AAA 
(63) 
(n-29p) 


518 
. (75) 
(n-310) 


A82 
(78) 
(n-31A) 


538 
(83) 
(n-259) 


536 
(73) 
(n-250) 

> 


A52 
(60) 
(n-273) 


530 
(7A) 
(n"»281) 


A91 
(76) 
(n-230) 


• 

* ■ ^ .* 




A^33 
(56) 
:n=1086) 


A95 
(82) 
(n-1129) 


A68 
(81) 
(n-lpl5) 


d 

312 
-(87) 
(n-671). . 


505 • 
(81) 
(n-635) 


















V 


1 

t ■ 




■10 ; 






• 




"V. 

< 
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O ■ 

ERIC 



Appendl)^ E 



ADDTTIONAUDATA FROM THE SANTA BARBARA 
DEMONSTRATION PROGRAM IN READING 



With the addition of standard score means and standard deviations. 
Tables E-1 through E-4 present^ the same data as T-ables IV-5 through 
IV~8 respectively. For purposes' of comparison^ we also present cross- 
sectional data. Table- E-5 presents the standard score means, and their 
associated grade-equivalent . and percentile scores for all students 
tested at a given point. > 

' ■ ' • ' / \ 

I ! 



• Table E-\ 

MEANS AND GAINS OVER TWO TIME PERIODS IN THREE METRICS FOR^ 
-STUDENTS WITH AT LEAST THREE CONSECUTIVE TEST POINTS* . 



BY GRADE. SANTA BARBARA DPR, CTBg READING 



Sam ple 



Grade 7 (n-l02) ^ 
Standard, scores 
Standard deviation 
(Irade equivalent 
Percentile 

Grade 8 (n-107) 
Standard scores 
Standard deviation 
Grade equivalents 
Percentile . 





Means 




* Gai 




I 


11 


III 


IV ' ' 


L- V 








Fall to 


Fall to 










. .. LtXX'JL 


405 « 


453 


446 


48^ 


41 


(70). 


(68) 


(65) 






4.3 


5.6 


5.4 


1.3 ' 


1.1 


12 


20 


16 


8 


4 


449 


484 


473 




24 


(64) 


(60) 


(64) 




) 


5. 5 


6.5 


6.2 


1.0 


0.7 


16 


23 


16 




0. 


. Table E-2 









MONTHLY ACHIEVEMENT RATES IN GRa6e EQUIVALENTS FOR TWO TIME 
PERIODS FOR STUDENTS WITH AT LEAST THREE CONSECUTIVE .. 
TEST POINTS, BY GBlU)E, SANTA BARBARA DPR, CJBS READING 



Monthly Achievement Rates 



Sample 



Grade 7 (n=102) 
•Grade 8 -(n=107) 



'1 

Fall to 
Spring 

I 

1.9 
^1.4 



II 
Fall to 
>aU 

1.1 

0.7 



. V 



E-4 



/ 



Table E-:} 



MEANS IN- THREK MI'IJITCS TOR STUDENTS WITH AT LEA5?T FI VE 
CONSECUTIVE TEST FOTnTS—SANTA BARBARA UPR, CTBS TRADING 



Grades 7-8 (n«99) 
"Standard score 
Standard deviation 
Crade equivalent 
Percentile 







Means 






1 


11 


III 


IV 


V 


Grade 7 


Crade 7 


Grade 8 


Grade 8 


Grade 9 


Fall 




Fall 




Fall 


405 


454 


448 


^82 


473 


(71) 


(69) 


(64) 


(60) 


\ (62) 


4.3 


5.6 


5.5 


. 6.5 


6:2 - 


12 


. 20 


16 • 


22 


16. 



Table E-4 • 



MONTHLY ACHIEVEMENT RATES IN GRADE EQUIVALENTS FOR THREE 
^ TIME PERIODS FOR STUDENTS WITH AT LEAS't FIVE CONSECUTIVE * 
TEST POINTS— SANTA BARBARA DPR, CtBS READING . 



Sample 

Grades 7-8 
(0=99) 



I II 

Grade 7 Grade 8 

Fall to » Fall to 

Spring Sprlnfi 



Monthly Achievement Rates 
in ~^ 



1.9 , 



1.4 



IV 



Grade 7 Pall to 
Grade 8 Spring 

1.3 - 



Gra^e 7 Fall to 
Grade 9 Fall 



1.0 



Table E-5 



.CROSS-SECTIONAL MEANS AND STANDARD DEVIATIONS FOR ALL 
STUDENTS TESTED— SANTA BARBARA DPR. CTBS READING 

Grade '7 ^"^ Gra4e 8 " Gra de 9 

Sprint^ Fall Spring Fall 

Grades 7'-8 ' . ^ '» ■ " ' . 

. 405 452 446 483 472 

Standard devlat^lon (72) (68) (65) (60) (63) 

.Number . (n°116) (n=105) (n-111) (n«112) (n='U4) 



{ 
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i f) ^ 



' /Appendix F • * . ^ 

ADDITIONAL DATA TOOM THE OMAHA PUBLIC SCHOOL HILINC PROGRAM- 

Appendix F contains additional information on the public school 
students in the Omaha HILINC program. Table F^l describes ,the cohorts 
for which we have data by year and grade level. Each cohort represents 
a group of students and is assigned the number that was the student's 
grade in 1971-72. Thus, we have 2. years- qf data^for Cohprt 1, 3 years ^ 
of data for Cohort 2 and 4 years for Cohorts 3-6. • 
~ ■ ' \ " ■' • ' . « ... 

Table F-2 presents the grade-equivalent means, standard^ deviations 
and- the gains over two time periods for samples of students with at least 
three consecutive test-points by grade and cohort. The totals, which 
are averaged across school and cohort, are the means presented in 
Table IV-9. Table F-3 presents the monthly achievement rates over two 
periods for the same samples of students. The rates are baked on the 
means in Table F-2. The .totals_^for each grade are those p^aeuted in 
Table IV-10. ' . 

Tables F-4 and F-5 present idata for samples of students with at 
least five consecutive test points. Table F-4 contains means by grade 
and cohort and Table F-5 contains Average monthly gains by grade and 
cohort. The totals from Tables F-4 and F-5 are those presented in 
Tables IV-ll and IV-12, respectively, - 

For purposes of comparison, we also ' include a table of cross- 
sectional data. The means for all students tested at a given po^nt are 
p.resented by grade and cohort in Table F-6. . • ' ! 

• ■ . ■■ ■ ■ ' " ) '. 
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F-3 
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, Table F-1 

COHORT DESCRIPTION BY YEAR AND GRADE 
OMAHA HIUNC ' 



* Sample „1971-72 , 1972-73 



Grade 3 
Grade 4 
Grade' 5. 
Grade 6 
Grade 7 
Grade 8 
Grade 9 



Cohort 3 
Cohort 4 
Cohort 5 
Cohort 6' 



Cohort 2 
Cohort 3 
Cohort 4 
Cohort 5 . 
Cohort 6 



1973-74 

Cohort 1 
Cohort 2 
Cohort 3 
Cohort 4 
Cohort 5 
Cohort 6 



1974-75 

Cohort 1 
Cohort 2. 
Cohort 3 
Cohort 4 
Cohort 5 
Cohort 6 



1 



,ERiC-. 



F-4 



0 s' 



Tab It? •F-2 



MEANS, ST/]jNUARD DEVIATIONS AND GAINS OVER TWO TIME PERIODS 
. IN GRADE-EQUIVALENT YEARS FOR STUDENTS WITH AT LEAST " 
THREE CONSECUTIVE TEST POINTS BY GRADE AND COHORT 
OMAHA HILINC GATES-MacGlNITIE READING 



Means and' 
^tandgrd De viation s) 
1- " iT 111 



Gains 





Fall 


Spring 


.Fall 


Sprlnf^ 


Grade 3 






2.60 / 




■ Cohort 1 (n=152) • 


1.97 


3.05 


. 1.08 




(.45) 


(1.10) 


(.78) 




t'Oiiort I (n=90j 


2 .67 


3. 72 . 


2.82 


« 

1.05 




(1.57) 


fl 86^ 






Cohort 3 « (n«30) 


2. 20 


3.26 


3.54 


' 1.06 




( 71 \ 




^1* 84 ; 




Total Grade 3 


2.23 


;. 3.29 


2. 78 


. 1.06 








(.96) 




Grade 4 


• 








Cohort 2 (n=387) 


'2.54. 


' 3.60 


3. 16 


1.06 




(.65) 


. a- 04) 


(.88) 




Cohort 3 (n=2 76) , 


2.81 


3.63 


3.07 


0,82 




.(JL.13) 


(1.48) 


(.88) 




Cohort 4 (n=274) 


2.64 


3.51 


3. 31 


0.87 




(.67) 


(1.07) 


(1.13) 




Total Grade 4 


2.65 


3.58 ; 


3.18 


' 0.93 


. . (n=931) 


(.83) 


(1.19) 


. (.96) 




Grade 5 , 










Cohort 3 (n=401) 


3.25 


4.45 


4. .10 


1.20 




(.90) 


(1.30) 


(1.32) 




Cohort 4 (n=278) 


3.46 


4.52 


3.99 


1. 06 




(1.22) 


^1.62) 


(1.22) 




Cohort 5 (n=301) 


•3.09 


3.89 


3.92 ' 


.0,80 




(.83) , 


(1.14) 


(1.34) 




Total Grade. 5 


3.26 


4. 30 


4 . 01 . 


1.04 


(n=980) 


(.99) 


(l.:38) 


(1.30) 





VI V 

Fall to Fall to 
Fall 



0.63 



0.15 



1.34 



0*55 



0.62 



0.26 



0.67 



0.85 
0.53 
0,83 
0.75 



Table F-2 (Concluded) 



Means and 
(Standard Deviation s ) 



Galna 





1 


11 


III 




V 










Fall\to 


Fall to 




Fall 


Spring 


Fall 


Spring 


Fall 


Grade 6 














3.59 


. 4.81 


4.41 


.1.22 


0.82 




(.99) 


(1.36) 


(1.22:1 






Cohort 5 (n=136) 


4.08 


4.79 


4.45 


0. 71 


0.37 




(1.38) 


(1.64) 


(1.31) 






•Cohort 6 (n=88) 


'3. 78 


4.75 


4.38 


O.9I 


0.60 




(1.02) 


<1,29) 


(1.43) 




f 


Total Crade 6 


3.85 


4.78 


4.42 


0.93 ' 


0.57 


(n=316) 


(1.20) 


(1.47) 


(1.32) 






"\ 

Gradf 7 / 












Cohort 5 ^n^73)' 


4. 37 


5. 30 


>i.8A 


, 0.93 


0.47 


(1.19) 


(1.58) 


(1.34) 






Cohort 6 (n=55) 


4.33 


5.18 


5.10 


0.85 


0.77 




(1.32) 


(1.82) 


(1.50) 






Total Grade 7 


4.35 


5.25 


4.95 


0.90 


0.60 


(n=128) 


(1.24) 


(1.68) 


(1..41) 







Table F-3 



MONTHLY ACHIKVE^tENT RATHS IN GRADE EQUIVALENTS FOR TWO TIME PERIODS 
. FOR STUDENTSi.WlTM AT LEAST THREE CONjSECUTIVE TEST POINTS BY 

GRADE AND COHORT 
,^ .OMAHA HILINC GATES-MacGINITIE READING 



Monthly Achievement Rate 
I II' 
Fall to Fall to 
^ Sample Spring . Fall - _ 



Gradp 3 

Cohort 1 (n-l52) 1.5 ' 0.6 

Cohort 2 (n=90) , . 1.5 0.2^ 

Cohort 3 (n=30) - 1.5 ■ 1.3 

Total Grade 3 (n=272) " 1.5 0.6 

Grade A 

Cohort 2 (n=387) 1.5 0.6 



Cohort 3 (n=270) \ 


1.2 


0.3 


Cohort A (n=27A) 


1.2 


0.7 


Total Grade A (n=931) 

-J 


1.3 


0.5 

» 


Grade 5 

Cohort 3 (n=AOI) 


1.7 . 


0;8 


Cohort A (n-278) 


1.5 ■ 


0.5 


, Cohort 5 (n=3dl) 


1.1 


0.8 


Total Grade 5 (n=980) . 


> 

1.5 


0.8 


Grade 6 
V Cohort A (n=92) 


« 

1.7 


0.8 


Cohort 5 (n'=136) 


■ 1.0 


0,A 


Cohort 6 (n'88) 


l.A 


0.6 


Total Grade 6 (n=316) 


1.3 


0.6 


Grade 7 

Cohort 5 (n=73) 


1.3 - 


0.5 


Cohort 6 (n=55) 


ir2 f 


0,8 


Total Grade 7 (ri=128) 


1.3 


0.6 



Tabl*j F-4 

MEANS AND STANpARU DEVIATIONS IN GRADE-EQUIVALENT YEARS 
Ft)R STUDENTS WITH AT LEAST FIVE CONSECliflVE 

TEST POINTS BY GRADE AND COHORT 
* OMAHA HILING ,GATES-MacGINITIE READING 



Sample 



Grades 3-4 
- Cohorr 2 (n»65) ' 
Means 

Standard 4^wlation 

Cohort 3 (a-22) 
Menus 

Standard deviation 

Total Grades 3-4 (n-87) 
Mt^aus 

Standard -deviation 

\ 

Grades A- 5 

Cohort 3 (u=178) 
• Means ' 
Standard deviation 

Cohort A' (n=lA6) 
Means 

Standard deviation 

Total Grades A-5 (n=32A) 
Means ' 

St?andard deviation 

Grades 5-6 

Cohot't A (n=50) ^ 
^ Means * y 

Standard d&vlation - 

• Cohort 5 (n=80) * ' 
Means 

Standard deviation 
Total Grades 5-6 (n=130) 

Standard deviation 



I 

Fall 



2.73 
(1.67) 

2.12' 
(0.60) 

2.57 
(1.A9) 



2.83 
(1.13) 

2.66 
(0.63) 

2.. 75 
(0.9A) 



3.39 
(1.A3; 

3.11- 
(0.92) 

3.22 
(1.15) 



ri 

Spring 



III 
Fall 



3.80 •^-a-:88 

(1.96) . (0.67) 



3.10 
(0.82) 

3.62 
(1.77) 



3.65 
.(1.A9) 

3. AL 
(0.96) 

3.5A 
(1.28) 



A. 33 
(1.86) 

A. 02 
(1.16) 

A.IA 
(1.^7) 



3»38" 
(1.8A) 

3.06 
(1. 10)^-(1.A2) 



IV 



3.99 
(1.11) 

A. 51 
(2.07) 

A;12 



3. OA 
(0.8A) 

3.19 
(0.92) 

3. 11 
(0.88) 



.011 



3. 

(l.CTl) 



A. OA 
(1.32) 

3.86 
(1.22) 



A. 29 
(1.28) 

A.2A 
(1.19) 

A. 27 
(1.2A) 



A. 89 
(l.AO) 

A. 76 
(1.58) 



A. 8^ 
(1.51) 



V 

F9II 



3.51 
(0.90) 

.3.59 
(0.99) 

3.53 
(0.92) 



3.85 
,(1.17) 

3.89 
(1.12> 

3.87 
(1.15) 



A. 29 
(1.16) 

A.A8 
(1.30) 

A.Al 
(1.25) 



Grades 6-7 ' ; 
Cohort 5 (n=A5) . 

Means A. 08 A.^1 A. 52 5.AA 

Standard. deviation (1.19) (1.58) (1.07) (i.A7) 

Total Grades 6-7 (n=A5) 

Means A. 08 A.91 A. 52 5.AA . 

Standard deviation (1.19) (1.58) (1.07) (1.47) 



5. OA 
(1.25)^ 

5. OA 
(1.25) 



,* TableHNF-5 • ' , " 

MONTHLY ACHIEVElMKNT RATES IN GRADE EQUIVAL^INTS FOR THREE TIME -PERIODS 
. ^OR STUDENTS.WITH AT LEAST FIVE CONSECUTIVE TEST -POlNTrBY 
GRADE AND COHORT-OMAHA .HU. INC, GATES-MacGINITIE READING 



Monthly Achievement. Rates 
1 'II , III IV^ 

Fall to Fall to Tall'l . -Fall'i 



San>p 1< 



Grades 3-4 

Cohort 2 .(n="67) , 

Cohort 3 (n=22) 

Total Grades' 3-4 (n=87)" 

Grades 4-5 

Cohort. 3' (n=178) . ' 

Cohort 4 (n=146) 

Total Grades 4-5^ (n=324) 

Grades 5-6 

Cohort 4 (n=50) 

Cohort 5 (n=80) ' 

Total Grades 5-6 (n=I30) 

Grades 6-7.. 

CohoPt 5 (n=45) 

Total Grades 6-7 (n=45) 



Spring 
Year 1 ., 



1.5 
1.4 
1.5 



1.2 
1.1 
1.1 



1. 3 
1.3 
1.3 



1.2 
1.2 



Spring 
Year 2 



1.6 
,1.6 
1.6' 



1.8 
1.5 
1.7 



1.9 

,1,0.. 
1.4 " 



1.3 
1.3 



to 

Spring 2 

0.7 
1.2f 
0.9 



0.9 
0.9 
0.9 



0-9 
1.0 

0.9 
r 



0.8 
0.8 



' to 
fall 3 

0.4 
0.7 
0.5 



0.5 
0.6 
0.6 



0.5 
0.7 
•0.6 



0.5 
0.5 



- . Table F-6 

CROSS-SECTIONAL MEANS AND STANDARD ^JATlOIJS IN GRADE EQUIVALENTS FOR ALL STUDENTS TESTED 

OMAHA HILINC, GATE^^-MacGINIXIE READING - 



Grade 3 



Cohort '1 - 
Means 

Standard deviations 
Number 

"Cohort 2 
Means 

Standard deviations 
Number 

Cohort 3 
Means 

Standard deviations 
Number 

Cohort 4 
Means 

Standard deviations 
Number 

Cohort 5 * 
Means 

Standard deviatil^ns 
Number 

Cohort 6' 
Means 

Standard devifit ions' 
Number.. 



Gr^de A 



Fall ^ SprlnR Fall Spring 



Grade 5* 



Grade 6 



Grade 7 



2.02 
(0.52) 
n"220 

2.86 
(1.74) 
n-117 

2.27 
(0.65) 
n=47 



PaJ^l Spring Fall Spring Fall . Spring 



3.U 
(1.13) 
^ n*^187 

4.09 
(2.07) 
n=117 

3.22' ■ 
(0.92) 
n?45 



2.79 
(0.93) 
0=784 

2.54 
(0.65) 
n-633 

3,06 
(1.46) 
n=435 

2.69 
(0.72) 
n»475 



3.62 
(1.27) 
n=564 

3.95 
(1.87) 
n-435 

3.53 
(1.10) 
n-432 



3.49 
(1.27) 
n"»644 

3.27 
(0.96) 
0=663 

3.61 
(1.41) 
ni5X9 

3.18 
(0.86) 
n«516 



4.44 
(1.35) 
n-559 

4.69 
.(1.81) 
n=519 

3.97 
(1.17) 
n»491 



4,30 
(1.72) 
n"»666 

4.08 
(1.27) 
n=529 



5.09 
(1.60) 
n-473 



4.37 .' 4.99 

(1.75) (2.00) 
n«566 n»566 

4.07 . 5.16 

(1.19) <1.46) 

"nw-499 n-429 



4.53 
(1.42) 
n-168 

4.40 
(1.46) 
n»249 

4.50 
(1.44) 
n-249 



5.26 
(1.82) 
n-15i 

5.40 

(r.90) 

n-249 
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/ Appendix G * • * 

DATA FROM OMAHA NONPUBLIC SCHOOL HIUNC PROGRAM 

Appendix G .contains the data for tb^onpublic school students In 
Omaha's HILINC reading program..^ These tables corresponcf to those pre- 
sented in the text for the public school students, llt'e cohpri; names are 
the same as those presented in Table F-1. 

The reading program was not begun in the nonpublic schools until 
.1972-73.. Therefore, Tables G-1 through G-5 span -fewer grades than the 
cotrespondlng tables for the public school student^. Table G-1 presents 
the means, standard deviations, and gains by grade and cohort for non- 
public school students with at least three consecutive test points. The 
achievement rates over two time periods for these student^ are given in 
Table G.-2. ^ ' ■ . 

Tables G-3 and; G-4 present data for t^e nonpublic school students 
with at least^ive consecutive test administrations. Table G-3 contains 
the means and standard deviations for these /students and Table G-4 pre- 
sents monthly achievement rates based on three periods of t;lme, ' 

Table G-5 presents means and standard deviations for all nonpublic 
scbool students tested at %ach point by cohort and gtade. 



lERIC 



G-3 



1 ^ 1 

^ ^ o 



Table G-1 



MJANS. 'STANDARD DKVIATIONS, AND GAINS OVER TOO TIME PERIODS IN 
i<;RADE-EQUIVAI.KNT YKARS for NONPUBLIC STUDENTS WITO AT LEAST 
THREE CONSECUTIVE TEST POINTS BY GRADE AND COHORT 
OMAHA HILINC GATES-MacGlNITIE " READING 



^eans and (Standard 
Devlatl(;iliiP 



I 



Gains 



Grade 4 
*l:ohort 2 (n-69) 
% 

cohort 3 (n=7.8) 

Totai Grade 4 .'(n-i4'7) 
* / ' . 

Grade 5 



II 



3. 34 4.40 
(.84) .(1.40) 

2.9-5 



(.76) 

3.13 
(.82) 



3.75 
(*1.00) 

4.06 
(1^4) 



\ 



III 
-Fall 

'4.09 
(1.29) 

• 3.77 
(,1.12) 

3.92 
(1.21) 



IV 
Fall to 
.Spring 

1.06 

0.80 

* 0.93 



\ 

Grade 6 



Grade 7 *" 



(a'=89) 


3.66 


5.01 


4.54 ■ 


1.35 




(.99) . 


(1.14) 


(1.13) * 




(n=88) 

• 


3.66 


4.75 


4. 74 


1.09 


• 


(1,01) 


(1,44) 


(1.36) 




5* (n=177) 


3.66 


4.88 


4.64 


1.22 




"(1.00) 

f 


(1.30) 


(1.25) - 




(n-49) 


4.33 


^ 5.13 ' 


• ,1 

4.76 


0.80 




. (a. 08) 


(1.28) 


(1.34) ; 




(;n=68) 


4.4,3 


. 5.14 


5.19 


0. 71 


* 


: (1.30) 

• ^ 


(1.53) ' 


(1.50) 




6 (n=117) 


'4.39 


5.14 


5.01 


0.75 




(1.21). 


(1.42) 

■ / ■ 


(1.44) ' • 





V 

Fall t9 
Fall 

0.75 

0.82 ^ 

0.79 

• 0.88 
1.08 
.0.98 

0.43 . 
0.76* 

0 

0.62 



Cohort 5 (n=38) 


4.73 


5.68 


5.60 


0.95 


0.87 




; (1.28) 


(1.88) ' 


(1.61) 






Cohort 6 (n='36) 


5'.21 


5.61 r 


5.68 ^ 


0.40 


0.47 




(1.46) 


(.2.21) 


(1.52). 






Total Grade ? (n=74) 


4.96^ 


. 5.65 


5.64 


-0.69 


0.68 




(1.38). 


(2.05) 


(1.56) 







\ 

'* Table G~2 * > ^ 

MONTHLY ACHIKVFKENT RATES 1*N GRADE EQUIVALENTS FOR WO TIME 
PERIODS FOR NONPUBLIC STUDENTS V/I'J'H AT LEAST THREE 
CONSECUTivE TEST POINTS BY GRADE AND COHORT 
OMAHA HILINC "GATES-MacGINITIE READING 

* 

. • Monthly Achievement Rate 



I II 

Fall-SprlnK Fall-Fall 

Grade A • 

Cohort 2 (n-69) 1.5 0.8 

Cohort 3 (n=78) ' 1.1 q.b 

Total Grade A (n«U7) 1.3 0.8 
Grade 5 

Cohort 3 (n=89) 1.9 0.9 

Cohort A (n=88) 1.6 1.1 

Total Grade 5 (n-177) 1.^7^ 1.0 

Grade 6 

Cohort A (ii=-49), ' 1.1 : O.A 

Cohort 5 (n=68) l.o 0.8 

Total Grade 6 (n-117) 1.1 0.^ 

Grade 7 • , * 

Cohort 5 (n=38) ' l.A 0.9 

Cohort 6 (n=36) , 0.6 . 0.5 

Tet>al Grade 7 (n^yA) 1.0 0.7 



Table G-3 



OkAllA IIILTNC GATKS-MacGlNrTfi^R^ADrNG MEANS AND STANDARD DEVIATIONS 
IN GRADE-EQUIVALENT \KARS FOR NONPUBLIC STUDENTS- WITH AT LEAST 
FIVE COl^SECUTiVE TEST POINTS BY GRADE AND COHORT 



1 



-GratleH 

Cohort 3 (n«59) 

Grades 5-6 ^ 
Cqhoft A Xn^yiY 

Grades 6-7 

Cohort 5 (n=28) 



/ 



1 

Fap 



2.84 
(.75) 



3.38 
(.98). 



4.36 
\1.35) 



II 

• Spring 



3.62 
(.93) 



4.49 
(1.29) 



4.88 
(1.31) 



III 
Fall 



3.61 
(.99) 



4.43 
(1.08) 



4.85 
^1.31) 



IV 
Spring 



V 

Fall 



4.96 ^ 4.45 
(1.16) % (1.12) 



5.14 
(1.22) 



5.67 . 
(2-. 08) 



4. 74 
(1.24) 



5.75 
(1.44) 



') Table G-A 



1 

OMAHA HILINC GATES-MacGINITIE READING MONTHLY- ACHIEVEMENT / 
RATES IN GRADE EQUIVALEOTS FOR THREE TilME PERIODS FOR^ ' / 
- NONPUBLIC STUDENTS WITIJ AT LEAST FIVE CONSECUTIVE / 



TEST POINTS BY GRADE AND COHORT 



Monthly Achievement 



Grades A~5 

' Cohort 3 (n-59) 
Grades 5-6 

' Cohort A (n=39) 
Grades 6-7 

Cohort 5 (n=28) 



Fall to Spring 
■ A B 
Grade 7 Grade 8 



1.1 



, 1.6 



0.7 



1.9. 



1.0 



1.2 



pent iy:e 
II ^ I 



II 



7 Fall to 7 Fall to 

8 Spring 9 Fall 



1'.2 



1.0 



0.8 



0.8 



0.7 



0.7 



' Table G-5 

OMAHA HILINC CROSS-SECTIONAL MEANS AND STANDARD DEVIATIONS IN GRADE EQUIVALENTS 
FOR GATES-MacGINITIE READING FOR ALL NONPUBLIC STUDENTS TESTED 



3F 



3S 



Cohort 2 

Means 2.72 

Standard deviations (1.65) 

Number n-14 

Cohort 3 
Means 

St/kndard deviations 
ber 



ohort 4 
Means 

Standard deviations 
Number 



Cohort 5 
Means 

Standard deviations 
Number 

Cohort 6 
M^ans 

Standard deviations 
Number 



3.19 
(2.14) 
n"14 



4F 



3.32 
(D.81) 



4S 



r 



4.37 
(1.37) 
n-93 



5F 



3.28 ■3.91 
(1.21) tl.51) 
n-133 :Vn-132 



4.08 
(1.13)' 
n=144 



3/90 
n.41) 

n-13, 



5S 



6F 



r 



4.02 
(1.30) 
n-150 



6S 



5.40. 
(1.36) 
n-135 



5.27 
(1.73) 
n=l49 



4.65 
(1.35) 
n-146 



' 4.76 
(1,42)- 
n^l4 



4.83 
(1.62) 
n=»167 



5.63. 
XI. 84) 
n=113 



5.67 
(2.10) 
n»167 



7F 



4.94 
(1.64) 
n-84 



A. 93 
(1.45) 
n-95- 



6.08 
(1.79) 
n«77 



r • 



Appendix H . ' 
ADDITIONAL DATA FROM THE AlUM ROCK VOUCHER DEMONSTRATION 
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Appendix H 



ADDITIONAL, DATA FROM THE ALUM ROCK VOUCHER DEMONSTRATION 



Appendix H contain^ additional dat^ from\ the Alum Rock Vciucher 
Demonstration., Table H-1 describes the cohorts for which we have data 
by year and grade level. From the table it can be seen that we have 2 
years of data from Cohorts 0 and 6 and 3 years of data from Cohorts 1-5. 

Table H-2- preserits.> the meantf ' and gains in standard scores and 
grade-equivalents for samples of students with at least three consecu- 
.tive test points by grade and cohorts. The totals for each grade are 
the means presented in Table VI-13. Table, H-3 presents the monthly 
achievement rates in grad^ equivalents for two time periods for the same 
•samples of students. The rates for each cohort and grade are based on 
the means in Table H^2. The totals. for each grade are those presented 
in Table VI-14. 

Table H-A presents the means in standard acores and grade equiva- 
lents for samples of students with at- least five consecutive/ test points 
by grade aiS cohort.- The grade-equivalent means are the means liresented 
in Table VI-15. 



Table li-l 

ALU^ ROCK VOUCHER DEMONSTRATION COHORT 
DESCRIPTION BY YEAR AND GRADE 

* « 

- - Year 

Grade 1972-73 1973-74' , 1974-75 



1 


Cohort 


1 


Cohort 


0 






-2 >• 


Cohort 


2 


Cohort 


1 


Cohort 


0 


3 


Cohort 


3 


Cohort 


Z 


Cohort 


1 


4 


Cohort 


4 


Cohort 


3 


Cohort 


2 


5 


Cohort 


5 


Cohort 


4 


Cohort 


3 


6 


Cohort 


6 


Cohort 


5 


Cohort 


4 


7 






Cohort 


6 


Cohort 


5 



Tabljt H-2 

ALUM^KOCK VOUCHER DEMONSTRATION MAT READING. MEANS AND GAINS IN TWO 
METRICS OVER TWO TIME PERIODS FOR STUDENTS WITH AT LEAST 
""THR^: CONSECUTIVE TESj* POINTS BY GRADE AND COHORT 



Means 



Gains 



. Sample 



Grade I 

. Cohort 0 (n-A5I) 
*. Standard score 
. Standard deviation 
Grade equivalent 

Cohort I (n-2l4) 
Standard deviation 
Standard score 
Grade, equivalent 

Total Grade I (n-665) 
. Standard score 
Standard deviation 
Grade equivalent 

Grade 2 

Cohort 1 (n«37l) 
Standard scote 
Standard deviation 
Grade equivalent 

Cohort 2 (n-211) 

A. 

Standard score 
Standard deviation 



II 



Fall Sprin p 



24.7 
(7.4) 
1.3 

28.3 
(6.4) 
1.4 

25.9 
(7.3) 
1.3 



37.8 
(11.3) 
1.8 

.39.9 
(11. 1) 




Grade equivalent 1,9 

total Grade 2 (n-Si82) 

Standard scor^ *\ 38.6 

Standard deviation (ll»3) 

Grade equivalent 1,8 

Grade 3 ^ ' ^. . \ 

Cohort 2 (n«480) 

Standard score 49,4 

Standard deviation (12.9) 

Grad^ equivalent 2.4 

Cohort 3' (n-301) 

Standard score . - , 49,7 

Standard deviation (12^3), 

Grade equivalent : i.5 

Total Grade 3 (n-7§l) : ' 

Standard score 49,5 

Standard Deviation (12.7) 

Grade equivalent 2.5 



. 35.3 
(8:8) 
1.6 

37.5 
(7.8) 
1.8 

36.0 
(8.6) 
1.7 



48.5 
(11.9) 
2.4 

49.4 
(11.2) 
2,4 

48.8 
(11.6) 
2.4 



56.6 
(11.8) 
3.1 



'56.5 
(12.8) 
3.1 

■* ■ 

56.6 
(12.2) 
3.1 



36.2 
(11.2) 
1.7 

37.8 
(11.3) 
1.8 

36.7 
(11.2) 
1.7 



50.9 
(11.6) 
2.5 

50.0 
(13.8) 
2.. 5 

50.6 
(12.4) 
2.5 



57.2 
(11.5) 
3.1 . 

57.2 
3.1 

57.2 
ri2.7) 
3.1 



.6 



0.3 

9.2 
0.4 

10. 1 
0.4 

10.7 
0.6 

9:5 

10.2 
0.6 

7.2 

6.8 
0.6 

7.1 
0.6 



V 

Fall to 
Fall 



U.5 
0.4^ 

9.5 
0.4 

10.8 
0.4 

P, 

13.1 
0.7 

10. 1 
0.6 

12.6 
0.7 

.7.8 
0.7 

7.5 

0.6 • 
7.7 

\ 

9 

0.6 



/ Table IN2 (Cbnttnued) 



Means 



\ ' Sample ; ^ 

Grade 4 

CohoVt 3 (n-'SSi) 
i^t^ndard sc6te 
Standard deviation^ 
Giracle equivalent <^ 

Cohort ^U (n«279) ^ 
** Standard score 
Standafrd ^jevlatlon 
Gradi? equivalent 

Total Grade 4 (n^832) 

Standard /score 

. Standard 'deviation 

* Grade . Equivalent 
«♦ 

Grade. 5. 

Cohort. 4 \n«541) 
Staadgrd score 
■ ; Standard deviation 
Grade equivalent, 

Cohort 5 (n«301) ' ^ * 
Standard scdre 
Standard deviation ; 
.Grande equivalent 

/Total GradQ 5 (n-842) 
Standard score 
. .Standard dejyiatton 
GraBe equivalent 

Grade 6 • ^ 

. Cohort 5 (n»533) ' * < 
Standard score ' 
Standard deviation 
Grade equivalent —* 

Cohort 6 (n-195) 
^ Standard ^5c6re 
Standard deviation 
Grade equivalent 

Total Grade 6 (;n-728^ 



.55.1 

'n2.;6.) 

2.9' 

5.5.9. 
(12.6) 
2.13 • 

54.7. 
(12.6) 
' 2,9 



, 61.9 
(13.4) 
'3.6 

61.5 
(13.1) 
3.6 

61.8 
(13.3) 
3.6' 



StaT^dard- score 
Standard, devlatidtt 
Grad^ equivalent 



4.2, 

68.4 
• <13.'2) 
.4.4 

. 6'7.7 
cV(.13.8) 
4.5 



, II -'kij 

Spring Fal,i 



61.3 .• 61.0. 
. (t2.6) • (13.^2) 
3.5 • 3.5. 



.60.0 61. s2 
,(13.5) '(14.0)' 



Gains 



3.4 



3.5 



60.9 61.1 
(12.9) U3.'5) 



3.5 



.3^5 



66.8 67.2 
(14..2) ,(15.2) 



4.2 



4.2 



65.8 65.3 
(13.8) (14.3) 



4.1 



3,9 



,66.5 66. 5> 
(14.1) (14.9) 



4.2 



67.2 . 71.4 
(14.0) (14.&) 
• 4.8 



*4.2 



73.1 
(14.9) 
4»9. 



' IV . 
Fall to 
Spring 



6.2 ' 

6.6 ■ 



V 

Fall to 
Fall 



5.9 
0.6 



6.1. ' ^ 7.3 
0..6 0.7 



6.2 • 

^ # - 

0.6 

*.'" •■ v 

0.6 

4.3 
O.57 

0.6 - 

4.2 
0.6 • 



.73.8 - 74.2 . 4.9 

(13.6) \ (13.4) . . . , 

5.0 ' 5.0 . .r- '-o.e 

. \ \ .1 

72.0 \ 73,4 . ' 4.3' 
.(14,5) ' (14.5) 

4.8 , 4,9 0.5 



6.4 
0.6 

"5.3 
0.6 

>3.8 
.0.3 • 

4,7 
0.6 . 

'■5.9 
0.7 

5.3 
0.6 

'5.. 7 
0.6 



Table 11-2 (Concluded) 



Meaga 



Gains 



Samp I e 



Grade 7 

Cohort 6 (n-605) 
Standard score 
Standard deviation 

> « 

Grade equivalent 

Cohort T (n-208)^ 
Standard score 
Standard deviation 
Grade equivalent 

tj^ , Total Grade 7 (n-813) 
Standard score 
. Standard de<riatlon 
Grade equivalent 



75.5 
(IA.4) 
.5.3 

78.9 
(13.f) 
5.7 



II 



80.5 
(14.7) 
.6.0 

83.* 
(13.4) 
6.4 



76.4 81.2 
(14.2I_. a4.4)_ 
5.3 6.0 



III 



Fall Spring Fall 



81.4 
(15.2) 

85.0- 
(14.6) 
6.7 

82.3 
X15.1) 



' IV 
Fall to 
Sprint^ 



5.0 

0.7. 



0.7 
4.8 



V 

Fall to 
Fall 



5.9 
0.7 



6.2 / 0.7 ^ 



6.1 
1.0 , 

5.9 
0.9 



\ 



# 



;ERjcva:.|g;.'| 



r 



H-7 



Talilo H-3 



ALUM ROCK VOUCHER DKMONS TRATION MAT READING MONTHLY ACHIEVEMENT RATES 
IN GRADE EQUIVALENTS FOR TWO TIME PERIODS FOR STUDENTS WITH AT 
LEAST THREE CONSECUTIVE TEST POINTS BY GRADE AND COHORT" 



..•ToCal 665 . 0.6 0.^4 - , * 

■ , . . . ..... i.'.r ■ . * • 



0 



2 - . 211 0/8. . - .0.6. 

2 ' 480 . "1.2 • • ■ 0.7- -K -v^ 

• 3 , ■ -ioi •■ 1.0 \ -5,0.7 . 

T^tai ... ' 781 :,v:: • '. L(l '■; : ^ o;?,; .> ■ 



3 553 ■ ■■ 1.0 - . " 0.6 

., A - 279^^ ;/ 1.0 : . 0.8 

TolaL: . -832 ■ ; . 1..0*V ' 0C7 



. • ^ 541 1.0 . " • * . 0.6r 

.5 30r' • 1.0;.; . -0.4 

Total ' 842 , -l V . . • „: '' 0.'5 . o 

• ■ ■' ... 

6. -5 533 0.7' ' ' . 0.> 

' ■ 6 195 , ■ 1.0 ; 6.7 

, 'Tol^l ^ 728 ^ % 0.8 •; .; V 0,7 

* * * * **. ■ ■ . ^ ■ „ 

7 ^ " 6 ' ^ 605 1.2 ' *%7 

■> ■■■■ 208 1.2 , . ' iTo; 

^.t, , Total- , 8.13 l.Z " ■ 0.9 



Table H-A .. . 

AI.UM ROCK VOUCHER DE^^ONSJ•RATION MAT READING MEANS IN 
TWO METRICS FOR STUDENTS WITH AT L^AST 
FIVE CONSECUTIVE TEST POINTS 







Grade 




Fall 


Spring 

.• 38.0 
.(8.3) 
1.8. 


Fall 


Spring 

47.6 
(11.1) 

■2.3^ 


Fall 




-2. 

0 


(n«147) 


^s 

(SD) 

. GE 


28.7 

(^.9) 

.1=.4 


■ r 

37.6 
(11.2) 
1.8 


50.7 
(11.6) 
• 2.5 


2- 


-3 


(n-147) 


SS 
(SD) 
GE 


40. 5 
(11.7) 
1.9 


49.8 
(11.5) 
2.5 


50.6 
(13.7) 
^ 2.5 


^ 5f/5 
(12.1) 
3.2 


58.1 
(11.7) 
. 3.2 


3- 


-4 


(n=193) 


SS 
(SD) 
GE 


50.0 

(12 
■> 2.5 


57.1 
3.1 ' 


5/9' 
3.2 . 


63.4 ' 
(13.9) 
3.7 


63.0 
(14.7) 
3.7 * 


4- 


-5 


(n=194) 


SS 
(SD) 
GE 


54.9 
(13.2) 
2.9 


61.4 
(13.7) 
3.5 


61.8 
(14.2) 
3.6 


67.8 
(15.0) 
4.3 - 


67.2 
(16.4) 
4.2 


5- 


6 


(n=191) 
» 


SS 
.(SD) 

■ GE 


62. 7 
(13.6) 
, 3.7 


67.1 
(13.8) 


67.0 
, (14.1) 
4.2 


71.^i.- 
(15.4) 
4.6 


73.3 
(15.2) 
4.9 


6- 


7 . 


(n=136) 


SS 
(SD) 
GE 


70.8 
(12.8) 
4.6 


76.1 
(13.2) 
5.3 


H.2 
(13.5) 
5.3 


82.2 
(12.7) 
• 6.2 


81.4 
(15.1) 
6.0 



vX 



